modelcertify

Supply-chain certification for trained models.

A model is only as trustworthy as the run that produced it. modelcertify makes that run auditable and certifies it against a written policy.

• A run proof hashes every artefact the run produced (config, metrics, checkpoints, weights) and combines the hashes into a Merkle root. Change one byte of one artefact and the root changes.
• An attestation is a signed statement by an issuer that they vouch for a proof root.
• A certificate is a policy evaluated against a run's evidence: it records every gate, how each one came out, the proof root it is bound to, and the issuer who signed it.

The certificate never overstates. It carries the gate results, so a reader can see why something was certified, not just that it was.

Install

pip install modelcertify
pip install -e ".[dev]"           # from a checkout

modelcertify has no runtime dependencies: it is stdlib-only on purpose, so it can be dropped into an air-gapped review environment.

Relationship to iroh-tinker

modelcertify is a standalone product. It does not import iroh-tinker, and it does not need to: it certifies a run's artefacts, wherever they were produced.

The natural pairing is that an iroh-tinker run writes its artefacts (the launch spec, the metrics, the checkpoints) to a directory, and modelcertify proof --dir <that directory> turns them into a proof.

CLI

modelcertify proof --run-id run-1 --dir ./artifacts --out proof.json
modelcertify policy --file policy.json --json
modelcertify certify --evidence evidence.json --policy policy.json \
    --issuer scai --key "$MODELCERTIFY_KEY" --out certificate.json
modelcertify verify --certificate certificate.json --key "$MODELCERTIFY_KEY"

Every command accepts --json.

Exit codes: 0 success, 1 the run was not certified (or the certificate did not verify), 2 a usage or input error.

The signing key is hex. Pass it with --key or set MODELCERTIFY_KEY.

Evidence and policies

An evidence document is what a policy is allowed to look at:

{
  "proof": {"run_id": "run-1", "artifacts": [{"name": "model.safetensors", "digest": "..."}]},
  "metrics": {"eval_loss": 0.21, "eval_accuracy": 0.94},
  "provenance": {"dataset_id": "ds-1", "licence": "cc-by-4.0"}
}

A policy is a set of gates over that evidence:

{
  "policy_id": "baseline",
  "name": "Baseline release policy",
  "required_artifacts": ["config.json", "model.safetensors"],
  "gates": [
    {"name": "loss", "kind": "metric", "field": "eval_loss", "op": "<=", "threshold": 0.5},
    {"name": "dataset", "kind": "provenance", "field": "dataset_id", "op": "!=", "threshold": null}
  ]
}

Gate kinds:

Operators: >=, <=, >, <, ==, !=, in.

What it does not do

• It does not prove the code was correct, only that the artefacts are the ones that were certified.
• The signature is an HMAC, not a public-key signature: the verifier needs the issuer's key. That is deliberate for v1 (no key ceremony before there is a customer), and the interface is narrow enough to swap in Ed25519 later.
• A failed policy still produces a certificate. The point of a certificate is to record the outcome, including a refusal.

Tests

pip install -e ".[dev]"
pytest

Layout

src/modelcertify/
  proof.py       run proofs, artefacts, the Merkle root
  attestation.py signed statements about a proof root
  certify.py     policies, gates, evidence, certificates
  cli.py         modelcertify proof | policy | certify | verify

Licence

Apache-2.0.