proofbundle

AI eval results need receipts.

Turn an AI evaluation result into one portable, offline-verifiable receipt. It proves who signed these exact bytes and that nothing changed since — not that the number is true. Ed25519 + RFC 6962 Merkle, one file, no server, no network.

Scorecard 6.5/10 — what the four zeros mean, in one sentence each

Reviewing this for adoption? Start with the 30-minute adversarial audit path: docs/REVIEWERS.md.

What the Scorecard badge says, including the parts that are low

The badge is live, so it will move. Measured 2026-08-07, re-measured 2026-08-10 with identical per-check values (Scorecard v5.5.0 both times): 6.5 / 10. Ten checks score 10/10 — Security-Policy, Token-Permissions, SAST, Fuzzing, CI-Tests, Vulnerabilities, Dangerous-Workflow, Dependency-Update-Tool, Packaging, License. Four score 0, and rather than let you wonder, here is each cause in one sentence:

• Maintained (0/10) — the check wants sustained activity on the default branch over 90 days, and this repository is younger than that window. It resolves itself with time and is not worth chasing.
• CII-Best-Practices (0/10) — the OpenSSF Best Practices badge has not been applied for. The criteria were walked through honestly first: docs/openssf_best_practices_self_assessment.md.
• Contributors (0/10) — it counts contributors from two or more organisations. This is a one-person project, and the zero is an accurate description of that.
• Signed-Releases (0/10) — the check reads GitHub release assets looking for a signature file. Every version release (v*) is attested (SLSA build provenance over the exact built bytes, PyPI upload gated on a sha256 match), but that attestation lives in GitHub's attestation store and on PyPI — not next to the release, which is where the check looks. The release workflow now also places the provenance bundle next to the release assets; that takes effect with the next release, and already-published releases were not modified after the fact. Nothing is re-signed — an existing file is placed in a second location. Of the five releases the check reads, three are corpus-review pre-releases that carry no such assets either, so the number will climb only as new releases move through that window.

Three further checks sit in between: Code-Review 1/10 (most commits are not reviewed by a second person — structural for a single maintainer), Binary-Artifacts 9/10 (the deducted point is a checked-in wheel+sdist pair kept as a reproduction fixture in dist_final/, and one point is not worth rebuilding that fixture), and Pinned-Dependencies 3/10 / Branch-Protection 3/10, both measured and not yet addressed.

Publishing a middling number with its causes is the point. A project that sells evidence cannot withhold its own.

60-second try (offline)

pip install "proofbundle[eval]"
proofbundle demo   # honest receipt => OK, six tampers each => FAILED, sample swap caught

The demo runs entirely in memory and exits non-zero if any tamper slips through, so it doubles as a self-test.

# verify a real hosted receipt without writing any code — the verify runs fully offline:
curl -fsSL https://raw.githubusercontent.com/b7n0de/proofbundle/main/examples/example_bundle.json -o receipt.json
proofbundle verify receipt.json        # CRYPTO: OK   (exit 0 ok · 1 fail · 2 malformed · 3 policy)

Emit your own receipt, apply a trust policy, start from a shipped template, or run the Inspect-native path (METR Task Standard / UK-AISI ecosystem, mockllm, no API key): docs/DEMO.md · Inspect walkthrough docs/INSPECT_HAPPY_PATH.md.

The problem

Every AI eval number you read — a safety benchmark, a capability score, a leaderboard entry — is an unverifiable claim. You trust the lab. There is no portable way to check, offline, that a result was signed by a stated party, has not been altered, and covers the samples it claims.

proofbundle is that check: a small MIT-licensed Python tool (a compact, auditable trusted verify core that depends only on cryptography; the package installs one more hard dependency, the RFC 8785 canonicalizer rfc8785, used on the emit and canonicalization paths) that turns a result into a signed receipt anyone can verify from a single file. In plain terms it is the cash-register receipt of an AI test result: it shows who claimed the number and that nobody quietly changed it, not that the test was good. Without a receipt there is nothing to check at all.

What a receipt proves, and what it doesn't

This boundary is the point, not a weakness. A receipt makes a claim attributable, tamper-evident, and — with pre-registration and per-sample auditing — bounded and spot-checkable. Full detail: THREAT_MODEL.md.

How it fits together

(diagram renders on GitHub — view it there; PyPI shows the source)

flowchart LR
    H["eval harness<br/>inspect_ai · lm-eval · promptfoo · pytest"] --> A["adapter → signed claim<br/>salted commitments · provenance · samples root"]
    A --> R["receipt<br/>one portable file"]
    R --> V{{"proofbundle verify — offline"}}
    V --> C["signature · Merkle inclusion · SD-JWT/KB ·<br/>witness quorum · status list · sample openings"]
    C --> OK(["CRYPTO: OK / FAILED"])
    style V fill:#D6248A,stroke:#D6248A,color:#fff
    style OK fill:#D6248A,stroke:#D6248A,color:#fff

proofbundle is a practical, released, offline verifier — complementary to TEE and zero-knowledge approaches, not a replacement for any of them. The neighbourhood, honest about the line each one crosses that a receipt does not (maturity labels stated so nothing reads as a settled standard):

Tool-by-tool comparison: INTEROP.md.

What's in the box

Each line is a one-sentence summary; the linked doc carries the exact flags, exit codes and version history (see also CHANGELOG.md).

• Core — Ed25519 signature + RFC 6962 / 9162 Merkle inclusion, verified fully offline against a real Sigstore Rekor proof, so correctness is not self-referential.
• Eval receipts — a signed claim (metric ⋈ threshold, n, salted model/dataset commitments, assurance level, provenance) from your run. EVAL_CLAIM.md
• Selective disclosure — SD-JWT (RFC 9901) with Key Binding: prove a threshold while withholding the exact score (unsigned or unbound disclosures fail closed).
• Transparency-log interop — C2SP tlog-checkpoint / cosignature / .tlog-proof, with post-quantum ML-DSA-44 witness cosignatures and optional Token-Status-List revocation.
• Per-sample audit — an auditor challenges random indices (fresh nonce or public randomness beacon); 300 samples catch 1% sample-doctoring at 95% confidence, regardless of run size — a challenge the issuer chose itself does not give this guarantee.
• Pre-registration — proofbundle prereg <plan> commits to the protocol before the run, so best-of-many publishing becomes visible.
• Integrations — opt-in inspect_ai end-of-task hook, pytest plugin, and a Hugging Face Community Evals bridge. INTEGRATIONS.md
• External time anchors (beta, [anchors] extra) — optional evidence of when a receipt existed, from a party the producer does not control; RFC 3161 and OpenTimestamps built in, plus a bring-your-own-type interface. Trust comes only from the relying party. docs/ANCHORS.md
• Universal content root (jcs-sha256-v1) — SHA-256 over the RFC 8785 (JCS) canonical bytes of the full pre-signature statement, so a content root survives counter-signing and key rotation; cross-implementation interop proven. ADR 0002
• Decision & action-outcome receipts — a signed decision (verdict, policy boundary, digest-bound evidence, and what was not checked) and a separately signed outcome with role separation (executor ≠ decision maker) — never a claim that the decision was correct. decision-receipt.md · action-outcome.md

Install

pip install proofbundle                 # core: offline verify + plain emit (two deps: cryptography, rfc8785)
pip install "proofbundle[eval]"          # + eval receipts, prereg, and the demo (RFC 8785 JCS canonicalizer)
pip install "proofbundle[inspect]"      # inspect_ai adapter + hook
pip install "proofbundle[pq]"           # verify ML-DSA-44 (post-quantum) witness cosignatures

Requires Python 3.10+. The verify path never rolls its own crypto — Ed25519 comes from cryptography; Merkle hashing is RFC 6962.

Post-quantum posture (honest)

proofbundle is not "quantum-safe" as a whole. Its hash-based layers (SHA-256, RFC 6962 / 9162 Merkle, RFC 8785 canonicalization, and the OpenTimestamps / chia-datalayer anchors) stay secure — Grover only halves SHA-256's effective strength, leaving a ~128-bit quantum margin. The Ed25519 receipt signature (and the RFC 3161 anchor's classical TSA certificate) are quantum-vulnerable to Shor. The attack that matters is back-dated forgery, and the defense is a hash-based time anchor: it proves the original receipt existed before any such capability, so a forged receipt has no matching anchor. The witness side already carries post-quantum ML-DSA-44 (FIPS 204) cosignatures; a post-quantum payload signature is on the roadmap. Detail: docs/ANCHORS.md.

Cite this work

If proofbundle helped your evaluation pipeline, please cite it. Machine-readable metadata is in CITATION.cff. The archival software record is on Zenodo under concept DOI 10.5281/zenodo.21110642; the Technical Note (design write-up) under concept DOI 10.5281/zenodo.21230466, also linked from b7n0de.com/proofbundle.

Docs

Status, scope and roadmap

Beta, SemVer-committed, with a CI test suite behind a mutation gate + property-based parser fuzzing. Correctness is anchored to external RFC 6962 vectors and a real Rekor proof, not just its own bundles; releases carry PEP 740 / SLSA build provenance. It is not a log service, a full in-toto client, a TEE, a consensus network, or a compliance product by itself — it is the small, offline, standards-native receipt layer between them. Security policy: SECURITY.md.

Roadmap (stated honestly, not yet built): a post-quantum payload signature (today the post-quantum coverage is witness-side ML-DSA-44 only), and a CLI flag to select the content-root algorithm (jcs-sha256-v1 is the signed default). Already shipped at preview/experimental maturity (install extra [experimental], API/wire-format may still change): a TEE-attestation bridge (RATS/EAT, RFC 9334 + RFC 9711) making assurance_level = enclave_attested independently verifiable — docs/EXPERIMENTAL_ENCLAVE.md.

Contributing

See CONTRIBUTING.md and the Code of Conduct. Good first issues are labeled good-first-issue; security findings go through SECURITY.md. The verifier core aims to stay small, dependency-light, and correct.

License

MIT — see LICENSE.

_{proofbundle is part of b7n0de, Verified AI Work · b7n0de.com}