proofbundle

AI eval results need receipts.

Turn an AI evaluation result into one portable, offline-verifiable receipt. It proves who signed these exact bytes and that nothing changed since — not that the number is true. Ed25519 + RFC 6962 Merkle, one file, no server, no network.

The problem

Every AI eval number you read — a safety benchmark, a capability score, a leaderboard entry — is an unverifiable claim. You trust the lab. There's no portable way to check, offline, that a result was signed by a stated party, hasn't been altered, and covers the samples it claims.

proofbundle is that check. It's a small MIT-licensed Python tool (a compact, auditable trusted core, depends only on cryptography) that turns a result into a signed receipt anyone can verify from a single file — and it's honest about the line it does not cross.

60-second try (offline, no setup)

pip install "proofbundle[eval]"
proofbundle demo

You'll see an honest receipt verify => OK, then six independent tampers each verify FAILED, then a swapped sample get caught — all in memory. The command exits non-zero if any tamper slips through, so it's also a self-test. Full walkthrough: docs/DEMO.md.

# your own receipt, from a signed payload:
proofbundle emit --payload-file result.json --new-key signer.key --out receipt.json
proofbundle verify receipt.json        # exit 0 = OK, 1 = failed, 2 = malformed

What a receipt proves — and what it doesn't

This boundary is the point, not a weakness. A receipt makes a claim attributable, tamper-evident, and — with pre-registration and per-sample auditing — bounded and spot-checkable. Full detail: THREAT_MODEL.md.

In plain language

A proofbundle receipt is the cash-register receipt of an AI test result: it shows who claimed the number and that nobody quietly changed it afterwards. It does not show the test was good — the way a cash-register receipt does not show the meal was good — but without a receipt there is nothing to check at all.

How it fits together

flowchart LR
    H["eval harness<br/>inspect_ai · lm-eval · promptfoo · pytest"] --> A["adapter → signed claim<br/>salted commitments · provenance · samples root"]
    A --> R["receipt<br/>one portable file"]
    R --> V{{"proofbundle verify — offline"}}
    V --> C["signature · Merkle inclusion · SD-JWT/KB ·<br/>witness quorum · status list · sample openings"]
    C --> OK(["=> OK / FAILED"])
    style V fill:#D6248A,stroke:#D6248A,color:#fff
    style OK fill:#D6248A,stroke:#D6248A,color:#fff

What's in the box

• Core — Ed25519 signature + RFC 6962 / 9162 Merkle inclusion, verified fully offline. Checks a real Sigstore Rekor proof, so correctness isn't self-referential.
• Eval receipts — a signed claim (metric ⋈ threshold, n, salted model/dataset commitments, assurance level, provenance) from your run. See EVAL_CLAIM.md.
• Selective disclosure — SD-JWT (RFC 9901) with Key Binding: prove a threshold while withholding the exact score.
• Transparency-log interop — C2SP tlog-checkpoint / cosignature / .tlog-proof, with post-quantum ML-DSA-44 witness cosignatures. Optional Token-Status-List revocation snapshots.
• Per-sample audit — commit to every sample; an auditor challenges random indices (with a fresh nonce or a public randomness beacon, v1.9) and openings must bind to the signed root. With such an auditor-supplied or beacon-bound challenge, 300 samples catch 1% sample-doctoring with 95% confidence, regardless of run size — a challenge the issuer chose itself does not give this guarantee.
• Pre-registration — proofbundle prereg <plan> commits to the protocol before the run, so best-of-many publishing becomes visible.
• Integrations — opt-in inspect_ai end-of-task hook and pytest plugin (emit only when PROOFBUNDLE_EMIT=1 / --proofbundle), plus a Hugging Face Community Evals bridge. See INTEGRATIONS.md, or the end-to-end walkthrough docs/INSPECT_HAPPY_PATH.md — run an eval, get a receipt, verify it offline.
• External time anchors (v2.0 beta, the [anchors] extra) — an optional anchors[] layer that attaches external evidence of when a commitment or receipt existed, from a party the producer does not control. Two built-in types verify offline: RFC 3161 TSA tokens (against a frozen cert chain) and OpenTimestamps Bitcoin proofs (honest pending → confirmed lifecycle). A register_anchor_type extension interface lets a third party ship its own fail-closed type; two worked examples ship — a first-party chia-datalayer/v1 (offline Merkle inclusion of a canonical root under a published Chia DataLayer root) and a third-party markovian-provenance/v1 (a wallet-attributable, Bitcoin-anchored stamp). See docs/ANCHORS.md.

Docs

Install

pip install proofbundle                 # core: offline verify + plain emit (dependency-free)
pip install "proofbundle[eval]"          # + eval receipts, prereg, and the demo (adds an RFC 8785 JCS canonicalizer)
pip install "proofbundle[inspect]"      # inspect_ai adapter + hook
pip install "proofbundle[pq]"           # verify ML-DSA-44 (post-quantum) witness cosignatures

Requires Python 3.10+. The verify path never rolls its own crypto — Ed25519 comes from cryptography; Merkle hashing is RFC 6962.

Status & scope

Beta, SemVer-committed, with a CI test suite behind a mutation gate + property-based parser fuzzing. Correctness is anchored to external RFC 6962 vectors and a real Rekor proof, not just its own bundles. It is not a log service, a full in-toto client, a TEE, a consensus network, or a compliance product by itself — it is the small, offline, standards-native receipt layer between them. Security policy: SECURITY.md.

Contributing

See CONTRIBUTING.md and the Code of Conduct. Good first issues are labeled good-first-issue; security findings go through SECURITY.md. The verifier core aims to stay small, dependency-light, and correct.

License

MIT — see LICENSE.

_{proofbundle is part of b7n0de, Verified AI Work · b7n0de.com}