veridata-recon

PyPI PyPI downloads Python versions License

Verifiable Reconciliation Proofs for Python - powered by Rust.

This is not the veridata pandas-cleaning package. This library provides cryptographic data pipeline reconciliation using Merkle trees, Ed25519 signatures, and the VRP (Verifiable Reconciliation Proof) format.

What It Does

veridata-recon lets you mathematically prove that data made it from a source system to a sink system without:

  • Drops - records lost in transit
  • Mutations - records altered during transfer
  • Duplicates - records replicated unexpectedly

It generates cryptographic proofs (VRP documents) that can be verified offline by any party with the public key - no access to the original data required.

VRP pipeline flow

Installation

pip install veridata-recon

Current version on PyPI: 0.1.1.

Quick Start

import veridata_recon as vr

# Generate a random salt for this reconciliation run
salt = vr.generate_salt()

# Your source and sink records (e.g., from Kafka topic and Iceberg table)
source = [
    {"order_id": "1001", "item": "widget", "qty": "5", "status": "shipped"},
    {"order_id": "1002", "item": "gadget", "qty": "3", "status": "pending"},
    {"order_id": "1003", "item": "gizmo",  "qty": "1", "status": "shipped"},
]

sink = [
    {"order_id": "1001", "item": "widget", "qty": "5", "status": "shipped"},
    {"order_id": "1002", "item": "gadget", "qty": "3", "status": "pending"},
    {"order_id": "1003", "item": "gizmo",  "qty": "1", "status": "shipped"},
]

# Reconcile with cryptographic proof
result = vr.reconcile(
    source=source,
    sink=sink,
    identity_rule="composite:[order_id]",
    content_fields=["order_id", "item", "qty", "status"],
    salt=salt,
)

print(result["verdict"])        # "PASS"
print(result["matched_count"])  # 3

Detecting Issues

# Source has 3 records, sink is missing one
sink_missing = source[:2]

result = vr.reconcile(
    source=source,
    sink=sink_missing,
    identity_rule="composite:[order_id]",
    content_fields=["order_id", "item", "qty", "status"],
    salt=salt,
)

print(result["verdict"])      # "FAIL"
print(len(result["missing"])) # 1 - order_id 1003 dropped

Key Features

Hashing

# SHA-256 (default) or BLAKE3
digest = vr.hash_bytes(b"hello world")
digest_b3 = vr.hash_bytes(b"hello world", algorithm="blake3")

Fingerprinting

fp = vr.fingerprint(
    record={"order_id": "1001", "amount": "99.99"},
    identity_rule="composite:[order_id]",
    content_fields=["order_id", "amount"],
    salt=salt,
)
# Returns: {"id_hash": "ab12...", "content_hash": "cd34...", "fingerprint": "ef56..."}

Key Management

# Generate a new Ed25519 key pair
keys = vr.generate_keypair()
print(keys["public_key"])   # base64-encoded
print(keys["private_key"])  # base64-encoded - keep secret!

# Reload from private key
keys2 = vr.keypair_from_private(keys["private_key"])
assert keys2["public_key"] == keys["public_key"]

Proof Verification

# Verify a .vrp.json proof file offline
outcome = vr.verify_proof("path/to/proof.vrp.json", public_key_b64)
# Returns: "PASS", "FAIL", or "UNVERIFIED"

Use Cases

  • Data Pipeline Integrity: Prove Kafka->Iceberg pipelines don't lose data
  • Regulatory Compliance: Cryptographic evidence of data completeness
  • CI/CD Gates: Fail builds if reconciliation doesn't pass
  • Cross-Team Trust: Share proofs without sharing raw data
  • Audit Trails: Chain proofs for continuous monitoring

How It Works

  1. Fingerprint each record using salted, domain-separated hashing
  2. Reconcile source vs sink fingerprint sets
  3. Produce a verdict: PASS, FAIL, or UNVERIFIED
  4. Generate Merkle proofs for missing records (offline verifiable)
  5. Sign the entire proof with Ed25519

Performance

Built on Rust with zero-copy where possible. Handles millions of records efficiently thanks to the underlying veridata-core engine.

License

Apache-2.0

Project links

Data verified by PyPI on Aug 26, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for vaquarkhan from gravatar.com vaquarkhan

Credits

Author: Vaquar Khan

GitHub Statistics

Data verified by PyPI on Aug 26, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License (Apache-2.0)

Requires

Python >=3.8

Tags

data-reconciliation cryptographic-proofs data-integrity merkle-tree pipeline-verification data-quality vrp

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

veridata_recon-0.1.1.tar.gz (59.0 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

veridata_recon-0.1.1-cp312-cp312-win_amd64.whl (507.9 kB view details)

Uploaded CPython 3.12Windows x86-64

veridata_recon-0.1.1-cp312-cp312-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl (1.2 MB view details)

Uploaded CPython 3.12macOS 10.12+ universal2 (ARM64, x86-64)macOS 10.12+ x86-64macOS 11.0+ ARM64

veridata_recon-0.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (666.8 kB view details)

Uploaded CPython 3.9manylinux: glibc 2.17+ x86-64

File details

Details for the file veridata_recon-0.1.1.tar.gz.

File metadata

  • Download URL: veridata_recon-0.1.1.tar.gz
  • Upload date:
  • Size: 59.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for veridata_recon-0.1.1.tar.gz
Algorithm Hash digest
SHA256 d665b24d8eaa56e472710bb16c5ebdf985e76c548bafc120afd3b6ee0b1aff3d
MD5 e9a0a13d7ef0cd521e620cf0ed394477
BLAKE2b-256 fa15376cf00beae78c05276eccd0c589c71ffb03f63511e2126b85e1336a7419

See more details on using hashes here.

Provenance

The following attestation bundles were made for veridata_recon-0.1.1.tar.gz:

Publisher: publish.yml on vaquarkhan/veridata

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file veridata_recon-0.1.1-cp312-cp312-win_amd64.whl.

File metadata

File hashes

Hashes for veridata_recon-0.1.1-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 2f0d5b30fa565b1f6257d6d1668717ed84eb88852c9746e13f486a3ad66fe0bc
MD5 863a21efd8fbc05c9c3039c1f97ba9f6
BLAKE2b-256 452f2c341fa653ad2c12efa517d27ab19f5c22c26df16ec82acc11b6d2b83436

See more details on using hashes here.

Provenance

The following attestation bundles were made for veridata_recon-0.1.1-cp312-cp312-win_amd64.whl:

Publisher: publish.yml on vaquarkhan/veridata

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file veridata_recon-0.1.1-cp312-cp312-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl.

File metadata

File hashes

Hashes for veridata_recon-0.1.1-cp312-cp312-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
Algorithm Hash digest
SHA256 6c58704cd856b8074bfbb72b1d0df08aa34f258f386db5845f0413d4b423b0de
MD5 2b2bf31918456b804ccd322fafb032d0
BLAKE2b-256 098ccb44ac53022031de1210fe3532120b961103ffc3d8e15248f3c7fbf071c9

See more details on using hashes here.

Provenance

The following attestation bundles were made for veridata_recon-0.1.1-cp312-cp312-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl:

Publisher: publish.yml on vaquarkhan/veridata

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file veridata_recon-0.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for veridata_recon-0.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e7fe05b39e899f6f8060a1fb11345012cf860504ddd1704d2e9efb6b5f8539e3
MD5 7b7ac7d919f7ac6cd302742d5df560ce
BLAKE2b-256 1abc49fa58eab6c1ffa42be7e1d955dc0fad1a7bd6b301cc8a7ee45237e9cd34

See more details on using hashes here.

Provenance

The following attestation bundles were made for veridata_recon-0.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on vaquarkhan/veridata

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.1.1 This release

4 files

0.1.0

4 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page