---

The problem

The AI industry runs on vendor-reported numbers. The uncomfortable, well-documented facts:

• A large share of published model benchmarks is vendor-reported, never independently re-run — public analyses have found the majority of leaderboard entries with no third-party reproduction.
• Reproduction failures are not hypothetical: high-profile releases have shipped with benchmark scores that could not be reproduced when re-measured (claims of ~57% vs ~33% measured on real suites).
• Integrity signing (Sigstore/OpenSSF-style, Hugging Face model signing) solved one narrow problem: the bytes didn't change. It says nothing about whether a measured quality claim re-derives from evidence.

svc (Synapsek Verified Claims) is the referee for the second problem. It is a framework-agnostic, cryptographically verifiable layer that signs measured quality claims about any AI artifact or pipeline — and lets a buyer re-derive every number with one command.

What one command does

$ svc verify claim.json --trusted-pubkey trusted-issuer.pub --base-dir . --rerun
=== SVC VERIFY synapsek.claim/knowledge-encoder-dedup/v3b-seed1 ===
  [PASS] L1 authenticity: valid
  [PASS] L2 integrity: 10 files match
  [PASS] L3 evidence: 2 traces match
  [PASS] L4 soundness: 1/1 values recomputed
  [PASS] L4b rerun: 1 claims re-executed
  [PASS] L5 claims: statuses + recommendations re-derived
  claim dedup_spearman: value=0.4172424238214698 recomputed=0.4172424238214698 status=proven [OK]
RESULT: PASS — all layers verified; claims re-derived

That is the entire buyer workflow. Exit code 0 = every layer you asked for passed. What you have actually established depends on which layers ran — a PASS with --rerun is a different (much stronger) statement than a PASS without it. The table below says exactly what each layer does and does not establish.

The five layers — and what each one establishes

Layer	What the buyer checks	What a PASS establishes	A PASS does NOT establish
L1 authenticity	Ed25519 signature under your trusted key; issuer key == trusted key	Who signed this claim, and that nothing changed after signing	That the numbers are good or even true
L2 integrity	SHA-256 of every pinned file (weights, code, data, docs)	The pinned bundle is exactly what the issuer signed	That the evaluation was run the way the docs say
L3 evidence	SHA-256 + item count + Merkle root of every evidence trace	The per-item evidence is complete and unedited (one swapped row fails this)	That the evidence came from a real run
L4 soundness	Every claimed value recomputed from the pinned evidence via its documented derivation rule	The claimed number is supported by the pinned evidence under the documented rule. For field:/file:/attested: rules this means read and re-aggregated from verified inputs — it is not a regeneration of the underlying model predictions	That the predictions themselves were generated by the pinned code
L4b re-execution (opt-in --rerun)	Re-runs the issuer's pinned evaluation command in your environment; compares its output to the claimed value within tolerance	The pinned eval actually reproduces the claimed number when you run it	That the eval measures what it claims to (the script is visible and hash-pinned — you read it; that judgment stays with you)
L5 claims	Every proven/not_proven status re-derived from the recomputed value + documented threshold; recommendation sets re-derived from proven labels	The claim's pass/fail labels and "recommended for" lists follow from the numbers; a claim that over-claims FAILS	That the threshold itself was a reasonable bar

The read / recompute / re-execute distinction matters and is never elided: reading a number from a signed summary (L4 via file:/attested:) is weaker than recomputing it from row-level evidence (L4 via field:), which is weaker than regenerating the predictions by running the pinned eval (L4b). Each example's disclosures state which level applies to each of its numbers.

If the issuer claimed more than the numbers support, verification fails — at a specific layer, with the specific reason. That last property, anti-overclaiming by construction, is what makes this different from a signed eval dashboard.

You worry about…	The layer that stops it
claim edited after signing (value, status, threshold)	L1
rogue issuer self-attesting under its own new key	L1
weights / code / data swapped in the claim dir	L2
evidence trace edited, reordered, truncated	L3
claimed value not supported by its own evidence	L4
a claimed value that only looks supported because the threshold was checked against the claimed number instead of the recomputed one	L4 + L5
number not reproducible by re-running the pinned eval	L4b
"proven" claimed below the documented threshold	L5
a not-proven label quietly added to "recommended for"	L5
claim circulated that was never registered	--registry

Quickstart

pip install synapsek-claims

# get the shipped examples
git clone https://github.com/Synapsek-Labs/synapsek-verified-claims.git
cd synapsek-verified-claims

# the dedup example pins a 160 MB weights file as a release asset — download it once:
mkdir -p examples/pipeline-dedup/artifact
curl -L -o examples/pipeline-dedup/artifact/encoder.pt \
  https://github.com/Synapsek-Labs/synapsek-verified-claims/releases/download/v0.1.0/encoder.pt

# buyer — verify the light example (no dependencies beyond the package):
svc verify examples/cartridge-v11/claim.json \
    --trusted-pubkey examples/trusted-issuer.pub \
    --base-dir examples/cartridge-v11 \
    --registry registry/registry.json

# buyer — verify with full re-execution (needs the pinned env: torch, transformers):
pip install torch transformers
svc verify examples/pipeline-dedup/claim.json \
    --trusted-pubkey examples/trusted-issuer.pub \
    --base-dir examples/pipeline-dedup --rerun

Notes:

• --rerun is explicit consent to execute code (the issuer's pinned evaluation, hash-pinned and readable before it runs). Without it, any claim with a script: derivation rule fails L4 with a clear "not executed" message — nothing runs silently, ever.
• Re-execution never goes through a shell: the pinned command is split into an argv list and executed directly. It runs in the claim directory with a timeout and with proxy/network env vars stripped (--no-env-scrub to disable), and only after L1+L2+L3 all pass.
• The pinned rerun command is exactly what the issuer pinned (here: python3 …). On systems where python3 is not the real interpreter (e.g. the Windows Store alias), use WSL or provide a real python3 on PATH — the command is deliberately not rewritten per platform.

Machine-readable reports: svc verify ... --json. Exit code 0 = PASS, 1 = FAIL.

Issuing a claim

svc keygen --out-dir ./keys
svc attest skeleton.json --key ./keys/issuer_ed25519 --base-dir ./claimdir
svc registry add registry.json --claim ./claimdir/claim.json --issuer-id you

What a claim is

A claim is a signed JSON document:

• subject — what it's about: files (hashed) + evidence traces (hashed + Merkle-rooted, one JSON item per line);
• pinned — the environment, dataset, and config the measurement used;
• claims — each with a metric, a measured value, a documented threshold, a derived status (proven / not_proven), and a derivation rule telling the buyer how to recompute it;
• recommendation — labels the numbers support, and labels they don't;
• disclosures — honest limitations, in writing, including which verification level (read / recompute / re-execute) applies to each number;
• signature — Ed25519 over the canonical form of all of the above.

Derivation rules are a tiny grammar with no hidden code:

rule	recomputation	input constraint
field:<name>|mean / field:<name>|last	aggregate a field over the evidence trace	the trace must pass L3
file:<path>#<field>	read a field from a pinned JSON file	the file must be a manifest entry that passed L2/L3, and the path must stay inside the claim directory
script:<path>#<id>	run the issuer's pinned eval; compare id from its final JSON stdout	runs only with --rerun AND after L1+L2+L3 all pass; executed without a shell
attested:<json.path>	read a value from a pinned attestation document	the document must be a manifest entry that passed L2

Every file a derivation rule can read must be pinned in the manifest and hash-verified before it is read. Non-finite values (NaN/Inf) — in claimed values, recomputed values, or tolerances — always fail; they can never "prove" anything.

Two working examples (real, measured, verify PASS as shipped)

examples/pipeline-dedup — a knowledge encoder (2-layer transformer + Perceiver resampler, K=128 latents) claiming Spearman 0.4172 between latent similarity and token Jaccard over a pinned 100-doc holdout (4,950 pairs, row-by-row evidence). L4 reads the value from the Merkle-pinned summary (a third party has independently recomputed the Spearman from all 4,950 rows and matched it). --rerun goes one level further: it re-executes the pinned eval on CPU and reproduces the value bit-exactly (tolerance 1e-6, observed drift 0).

examples/cartridge-v11 — the Synapsek knowledge cartridge v1.1 wrapped through the same pipeline: 13 claims. The 4 v0 gist dimensions are recorded as not_proven — below the pre-registered bars (topic 22.67 vs ≥50, dedup 0.122 vs ≥0.4, retrieval 9.33 vs ≥25, summarize 0.123 vs ≥0.2). An honest claim is allowed to say no; the system enforces that it actually means it. The 9 QA-fidelity claims are proven: 6 are re-derived from pinned per-item QA traces (L4 via field:); the 3 cross-family 7B scores are carried from a pinned attestation document (L4 via attested: — the per-item traces for those live in that attestation's own run, disclosed in the claim).

$ svc verify examples/cartridge-v11/claim.json --trusted-pubkey ... --base-dir .
  [PASS] L1 authenticity: valid
  [PASS] L2 integrity: 1 files match
  [PASS] L3 evidence: 6 traces match
  [PASS] L4 soundness: 13/13 values recomputed
  [PASS] L5 claims: statuses + recommendations re-derived
  claim gist_topic: value=22.67 recomputed=22.67 status=not_proven [OK]
  claim qa_qwen_s1: value=0.097809 recomputed=0.0978094028382634 status=proven [OK]
  claim qa_7b_s3: value=7.55 recomputed=7.55 status=proven [OK]
RESULT: PASS — all layers verified; claims re-derived

Trust model

• Ed25519 over deterministic canonical bytes (sorted keys, compact JSON, UTF-8). Same byte format as the Synapsek v1.1 attestation engine — the two ecosystems interoperate.
• Only the public key ships with claims. Buyers trust a public key held out-of-band (examples/trusted-issuer.pub, or the v1.1 attestation bundle); a claim cannot choose its own trusted key.
• The registry (registry/registry.json) is a plain-JSON trust anchor: claim id → issuer key → claim hash. Portable today, a hosted endpoint later. See examples/TRUST.md.
• Every layer reports independently: a failure names the layer, the file, the claim, and the reason.

What this is — and is not

✅ Signs quality claims and proves they re-derive, at a stated level (read / recompute / re-execute)	❌ Not a model-signing replacement — integrity signing is a commodity and a good one; SVC is complementary
✅ Pins your evaluation; structurally hostile to fudging	❌ Not a benchmarking service — it cannot make a bad number look good
✅ proven = the recomputed value meets the documented threshold, and the claim says so	❌ Not a quality certifier — the threshold is the issuer's disclosed commitment; the buyer judges it
✅ Execution is consent-gated, manifest-pinned, and shell-free	❌ Not a sandbox — a script:/rerun command you run on your machine can do what any command can; read it first, run it where you'd run that eval

Roadmap

• claim schema v0.1, 5-layer verify, pinned re-run engine, registry, CLI
• execution consent gates, no-shell executor, manifest-pinned derivation inputs, finite-value guards (2026-10-17 audit fixes)
• two attested real-world examples, 29-test suite, Apache-2.0, CI
• public registry endpoint + claim URI scheme
• verifiable agent-execution records (tool calls, approvals, audit logs as evidence)
• batch verify; GPU/CPU cross-device tolerance policies

Security & contributing

Found a verification bypass? Read SECURITY.md before reporting. Contribution guidelines in CONTRIBUTING.md.

Design rationale and threat model: docs/DESIGN.md. Release/publish runbook: RELEASE-CHECKLIST.md.

Tests

python3 -m pytest tests/ -q    # 29 tests: roundtrip, tamper L1/L2/L3/L4b/L5,
                                # execution-consent gates, overclaim, NaN guards,
                                # manifest/path-escape guards, UTF-8, registry, canonical

License

Apache-2.0 © Synapsek Labs