ProofFrame

ProofFrame compiles strict data contracts into typed Arrow kernels. It scans record-batch streams, keeps evidence bounded, and fails before execution when a rule does not match the physical schema.

Version 0.5.0 is a beta release. Its release gates cover synthetic scaling and allocation contracts; the pinned 7,645,034-row Bitcoin comparison remains a dedicated-runner gate, not a published claim.

What 0.5 changes

• Contracts are a versioned, deny-unknown-fields syntax tree compiled against the Arrow schema.
• Numeric, timestamp, decimal, string, binary, and uniqueness rules execute through specialized kernels rather than per-row dynamic dispatch.
• Exact uniqueness, keyed diff, and leakage checks have explicit memory, temporary-storage, output, and sample limits. Known small inputs stay in memory; larger exact state spills as checksummed runs and partitions when its memory account is exhausted.
• pf-fp-v1 stays frozen for existing proofs. pf-fp-v2 adds a segmented encoder designed for prepared buffers and stable batch-independent hashing.
• Python receives native dictionaries and typed exceptions; the Rust scan runs with the GIL released. Current Pandas and Polars frames are consumed through Arrow C Stream without Python row materialization, including Arrow Utf8View and BinaryView columns.
• Evidence V2 binds dataset, contract, engine, limits, and result. Receipt verification reports cryptographic integrity separately from signer trust.

The Rust crate forbids unsafe code. PyO3 and Arrow's FFI remain dependency boundaries and are tested through wheel-level Python integration tests.

Install

pip install proofframe==0.5.0

Rust users can install the core without Python:

cargo add proofframe@0.5.0

ProofFrame supports Python 3.10–3.13 and Rust 1.85 or newer.

Check a table

import pyarrow as pa
import proofframe as pf

table = pa.table({
    "order_id": [101, 102, 102],
    "amount": [12.50, 8.00, -1.00],
})

contract = {
    "version": "proofframe.contract.v1",
    "columns": {
        "order_id": {"required": True, "not_null": True, "unique": True},
        "amount": {"required": True, "min": 0},
    },
    "max_findings": 20,
}

report = pf.check(
    table,
    contract,
    max_memory=64 * 1024 * 1024,
    max_temp=512 * 1024 * 1024,
    max_samples=20,
)

assert report["valid"] is False
assert report["violation_count"] == 2

violation_count is exact even when the findings sample is truncated. Missing required columns and incompatible rule/type combinations are rejected during compilation, before rows are scanned. Timestamp bounds accept signed integer ticks in the Arrow field's declared unit or offset-qualified ISO-8601/RFC 3339 strings such as 2026-08-15T12:30:00+03:00. String bounds are normalized to UTC and must map exactly to the field unit; ProofFrame rejects precision that would require rounding.

Pandas, Polars, PyArrow tables, record batches, readers, and Arrow C Stream providers are accepted. Known containers also provide exact row and logical-byte hints to the Rust engine so exact buffers and small diffs can be sized before scanning. Inputs that expose only a stream stay streaming; the CLI does not construct a full table for CSV or Parquet input.

Fingerprints

legacy = pf.fingerprint(table, version="v1")
current = pf.fingerprint(table, version="v2")

Fingerprint versions are separate protocols. Never compare a V1 digest with a V2 digest. V1 is the Python compatibility default for 0.5; the CLI defaults new work to V2:

proofframe fingerprint data.parquet --fingerprint-version v2

profile() is a compatibility operation and defaults to distinct="none". Exact cardinality is explicit and uses the bounded spill engine:

profile = pf.profile(
    table,
    distinct="exact",
    max_memory=64 << 20,
    max_temp=1 << 30,
    spill="auto",  # use "never" to fail instead of writing exact-state runs
)

Evidence and receipts

checked = pf.check_with_evidence(table, contract, max_samples=20)
report = checked["report"]
evidence = checked["evidence"]
keys = pf.generate_keypair()
receipt = pf.sign_evidence(evidence, private_key=keys["private_key"])
verification = pf.verify_receipt(receipt, expected_public_key=keys["public_key"])
assert verification["valid"]

check_with_evidence validates and fingerprints each Arrow batch in the same native execution. Evidence V2 separately binds canonical contract source (pf-contract-v1), compiled typed plan (pf-plan-v1), and Arrow schema (pf-schema-v1). sign_receipt(..., receipt_version="v1") exists only for migration; normal Python and CLI signing defaults to V2.

PII findings use keyed 256-bit fingerprints. The default scan key is random per run and unlinkable. For stable correlation, pass a URL-safe base64 32-byte fingerprint_key and a non-secret key_id; the key is never returned or embedded in evidence.

Exact diff with bounded output

result = pf.diff(
    before,
    after,
    keys="order_id",
    max_memory=256 * 1024 * 1024,
    max_temp=2 * 1024 * 1024 * 1024,
    max_samples=100,
    max_output_records=1_000_000,
    output="changes.jsonl",
    spill="auto",
)

Counts remain exact. In-memory examples are bounded by max_samples; complete change records can be written atomically as JSON Lines or Arrow IPC, up to max_output_records. With trusted row/byte hints, spill="auto" uses a conservatively budgeted in-memory path for small diffs and reports zero partitions. Larger or unknown streams use temporary partitions carrying a format version, schema digest, declared lengths, and a BLAKE3 checksum. spill="never" prohibits exact-state data spill and fails closed at max_memory; output files still use an atomic same-directory temporary file. Length limits are checked before allocation.

CLI

proofframe check data.parquet --contract contract.json --max-memory 256MiB --max-temp 2GiB
proofframe diff old.parquet new.parquet --key order_id --output changes.jsonl --spill auto
proofframe fingerprint data.csv --fingerprint-version v2
proofframe evidence data.parquet --contract contract.json --output evidence.json
proofframe sign evidence.json --private-key "$PROOFFRAME_PRIVATE_KEY"
proofframe verify receipt.json --expected-public-key "$PROOFFRAME_PUBLIC_KEY"

Exit codes are stable:

JSON output is emitted only after a successful operation. File outputs use a same-directory temporary file, flush and fsync, then an atomic replace.

Compatibility

validate() and profile() remain migration shims for the 0.5 line. validate() delegates to check() and no longer creates an implicit profile. New code should call check() and fingerprint() separately.

Release tags are publishable only after CI on that exact commit emits the release-evidence artifact. The Rust package path is checked with cargo publish --dry-run --locked before registry publishing.

See the 0.4 to 0.5 migration guide for API and serialized-contract changes. Rust API details are in README-crates.md.

Performance evidence

The release harness records raw samples, median, IQR, throughput, subprocess peak RSS, engine memory and spill counters, algorithm version, schema, compiler, CPU, and dataset SHA-256. It refuses to compare artifacts captured on different hardware, compilers, datasets, or fingerprint versions.

python benchmarks/release_gate.py \
  --rows 100000 --runs 5 --warmups 1 \
  --output target/release-gate-smoke.json

This command is a deterministic synthetic smoke test. The real Bitcoin fixture is identified by benchmarks/fixtures/bitcoin-7_6m-manifest.json; the data itself is not redistributed. Methodology and remaining stable-release gates are documented in docs/testing.md.

Development

cargo test --locked --all-targets --all-features
cargo clippy --locked --all-targets --all-features -- -D warnings
maturin develop --release --locked
python -m pytest -q

CI additionally runs Miri-compatible state-machine tests, three 60-second fuzz targets, release-mode allocation contracts, Python 3.10–3.13 on Linux/macOS/Windows, wheel smoke tests, and package checks.

Apache-2.0 licensed. Security reports follow SECURITY.md.