modelcertify

Supply-chain certification for trained models.

A model is only as trustworthy as the run that produced it. modelcertify makes that run auditable and certifies it against a written policy.

• A run proof hashes every artefact the run produced (config, metrics, checkpoints, weights) and combines the hashes into a Merkle root. Change one byte of one artefact and the root changes.
• An attestation is a signed statement by an issuer that they vouch for a proof root.
• A certificate is a policy evaluated against a run's evidence: it records every gate, how each one came out, the proof root it is bound to, and the issuer who signed it.
• An evaluation suite (P4.x) is a private set of scored cases. Running it produces a report whose suite digest is committed to the proof, so a policy can gate on exactly which suite was used without shipping the suite. The runner supports two modes: quick (deterministic subsample, for iteration) and full (every case, for the attestable result). Scoring is a dependency-free port of the decision-index primitives (accuracy, macro/binary F1, NDCG@10, MRR, chance baselines).

The certificate never overstates. It carries the gate results, so a reader can see why something was certified, not just that it was.

Install

pip install modelcertify
pip install -e ".[dev]"           # from a checkout

modelcertify has no runtime dependencies: it is stdlib-only on purpose, so it can be dropped into an air-gapped review environment.

Relationship to iroh-tinker

modelcertify is a standalone product. It does not import iroh-tinker, and it does not need to: it certifies a run's artefacts, wherever they were produced.

The natural pairing is that an iroh-tinker run writes its artefacts (the launch spec, the metrics, the checkpoints) to a directory, and modelcertify proof --dir <that directory> turns them into a proof.

CLI

modelcertify proof --run-id run-1 --dir ./artifacts --out proof.json
modelcertify policy --file policy.json --json
modelcertify certify --evidence evidence.json --policy policy.json \
    --issuer scai --key "$MODELCERTIFY_KEY" --out certificate.json
modelcertify verify --certificate certificate.json --key "$MODELCERTIFY_KEY"
modelcertify eval --suite suite.json --mode full --run-id run-1 --out certificate.json

Every command accepts --json.

Exit codes: 0 success, 1 the run was not certified (or the certificate did not verify), 2 a usage or input error.

The signing key is hex. Pass it with --key or set MODELCERTIFY_KEY.

Evidence and policies

An evidence document is what a policy is allowed to look at:

{
  "proof": {"run_id": "run-1", "artifacts": [{"name": "model.safetensors", "digest": "..."}]},
  "metrics": {"eval_loss": 0.21, "eval_accuracy": 0.94},
  "provenance": {"dataset_id": "ds-1", "licence": "cc-by-4.0"}
}

A policy is a set of gates over that evidence:

{
  "policy_id": "baseline",
  "name": "Baseline release policy",
  "required_artifacts": ["config.json", "model.safetensors"],
  "gates": [
    {"name": "loss", "kind": "metric", "field": "eval_loss", "op": "<=", "threshold": 0.5},
    {"name": "dataset", "kind": "provenance", "field": "dataset_id", "op": "!=", "threshold": null}
  ]
}

Gate kinds:

Operators: >=, <=, >, <, ==, !=, in.

What it does not do

• It does not prove the code was correct, only that the artefacts are the ones that were certified.
• The signature is an HMAC, not a public-key signature: the verifier needs the issuer's key. That is deliberate for v1 (no key ceremony before there is a customer), and the interface is narrow enough to swap in Ed25519 later.
• A failed policy still produces a certificate. The point of a certificate is to record the outcome, including a refusal.

Tests

pip install -e ".[dev]"
pytest

Layout

src/modelcertify/
  proof.py       run proofs, artefacts, the Merkle root
  attestation.py signed statements about a proof root
  certify.py     policies, gates, evidence, certificates
  cli.py         modelcertify proof | policy | certify | verify

Licence

Apache-2.0.