Merriv

An optimized model can match backend outputs and still become a worse release. Merriv catches the regression, records the evidence, and localizes the first bad build when the evidence is decisive—before a promotion controller acts.

Merriv is the vendor-neutral release-evidence layer for deployable AI models. It turns a change between model builds into a portable, content-addressed, and independently checkable Model Change Report.

Model Change Report is the portable evidence contract. Merriv is its reference implementation.

Status: Pre-alpha reference implementation · seeking three external release pilots, not claiming a standard or external adoption · roadmap

Named products in the diagram are interoperability examples, not bundled dependencies or endorsements. Verification covers declared integrity and conformance; it does not establish producer identity. The diagram has an editable draw.io source.

[!IMPORTANT] Replay of a real historical regression: llama.cpp issue #22544 identifies a first-bad commit where --tensor-type was ignored during quantization; merged PR #22572 fixed it. The source-anchored replay returns BLOCK on the two tensor assignments published upstream. It is a replay, not a fresh 27B model execution or a model-quality claim.

Quick start

Run the offline demo without cloning the repository:

[!NOTE] This quickstart intentionally produces a BLOCK report. The convenience demo wrapper exits 0 after writing the demonstration; merriv compare and the GitHub Action enforce the report and return exit code 2 for BLOCK.

uvx --python 3.13 --from merriv==0.1.0a3 merriv demo --output runs/quickstart

The declared rare slice regresses more sharply than the common slice. The command writes a compiled release plan, evidence manifest, Model Change Report JSON, Markdown, JUnit, and SARIF. It is a synthetic behavior demo, not adoption or empirical evidence. The command pins the published alpha instead of installing mutable main. See the full quickstart.

Already have retained Polygraphy results? Import them without first learning the recorded JSONL format:

merriv import polygraphy run-results.json \
  --baseline-runner onnxrt-runner \
  --candidate-runner trt-runner \
  --policy policy.yaml \
  --output runs/polygraphy-mcr

The importer uses Polygraphy's native comparator. --format normalized is a wiring/test interchange and is explicitly labeled as non-live evidence.

Why this exists

Model builds cross optimizer, compiler, runtime, hardware, registry, and CI boundaries. Each tool may produce a correct local answer while the release still lacks one portable object that binds the exact artifacts, evidence, statistics, policy, decision, and regression onset.

Merriv does not replace native tools:

Those are evaluation decisions, not deployment authority. A Model Change Report says whether its bound evaluation policy was satisfied. The consuming organization combines the verified report, producer identity, provenance, BOM, risk, and environment policy to make its separate ALLOW/DENY decision.

For prompts, RAG applications, or agent trajectories, use an application-evaluation tool first. For backend or layer debugging, use the native debugger first. Merriv starts where those results must become reviewable release evidence. The detailed boundaries are documented in when to use each tool.

Reproducible release regression

The CPU-only ONNX experiment exports one fixed model to FP16 and three real INT8 QDQ builds and evaluates 629 paired holdout cases. It intentionally changes the calibration range, so it is a controlled regression test—not the headline proof:

Pinned CI runs return WARN or BLOCK for the contracted-calibration builds as runner-specific quantization numerics move borderline cases within the declared accuracy ranges. A generated Model Change Report is authoritative for its exact artifact and runtime; only evidence that clears the full retained interval and Holm requirements is promoted to BLOCK.

For the retained NVIDIA scale-0.65 run, the critical-slice paired change is -12.77 percentage points with a raw 95% percentile-bootstrap CI of [-23.40, -4.26] points (n=47). That width is why reports now expose sample size, CI level, family-wise alpha, Holm-adjusted evidence, target power, and MDE. Merriv 0.1.0a3 evaluates non-zero matched-binary margins with Tango's score test and its inverted score interval; it never applies the continuous sign-randomization assumption to binary outcomes. In the retained NVIDIA scale-0.65 run, the critical-slice two-sided score p-value is 0.0224, Holm adjusted to 0.0447, with a 97.5% score interval of [-27.34, -1.88] points, so that rule is BLOCK. The reference row is declared context, not a candidate release verdict. Ordered localization still claims no first bad build because the balanced endpoint is WARN, not a decisive PASS; uncertainty is not silently converted into an onset. The example also records ONNX semantic diff, per-tensor numerical divergence, gate evidence, and executed bisect. Run it with:

[!NOTE] The retained ONNX evidence toolchain currently requires Python 3.11–3.13. Merriv's base package supports Python 3.14, but the onnx-demo extra intentionally does not install its older retained dependencies on 3.14.

uv sync --python 3.13 --frozen --extra onnx-demo
uv run --frozen python examples/onnx_quantization/run_demo.py --output runs/onnx-quantization

The NVIDIA vertical exercises the same boundary with ModelOpt, TensorRT, and Polygraphy on an RTX 4060 Laptop GPU. Its large retained evidence is distributed as a GitHub Release asset rather than stored in Git history; the repository keeps the small receipt, hashes, scripts, and reproducible case study.

Reproduce on your hardware

Independent reproductions across NVIDIA GPUs, TensorRT versions, operating systems, and other deployment runtimes are welcome. If you reproduce this release story on another system, open an External Reproduction report. Repository-owned reruns are not counted as external adoption.

Model Change Report at a glance

Version 0.4 binds:

• immutable baseline and candidate snapshot identities;
• executor, runtime, platform, and build provenance;
• paired metrics, uncertainty, sample size, and slice scope;
• the exact versioned policy and five-state evaluation decision;
• policy-wide Holm-Bonferroni correction, family-wise alpha, target power, and MDE;
• content-addressed evidence sets and supplemental evidence;
• a replay-stable evidence_id, decision-bound report id, and exact run_id;
• optional artifact diff, numerical diff, and first-bad-build evidence.

Any conforming producer may emit a Model Change Report without using the Merriv CLI. A consumer can verify and consume it without importing the merriv Python module:

merriv conformance producer examples/mcr_conformance
merriv mcr verify examples/mcr_conformance/full --strict

valid means the performed integrity checks passed. It does not mean the producer is authenticated, a transparency record was checked, an independent party reproduced the evidence, the model is safe, or deployment is authorized. Verification reports each trust dimension separately.

Project map

The base package has four runtime dependencies: httpx, pydantic, PyYAML, and typer. ONNX, demo, and integration toolchains remain optional.

Trust and attestation direction

Model Change Report content IDs establish integrity, not producer identity. The integration direction is therefore an in-toto Statement predicate that can be signed with Sigstore/cosign and carried by existing artifact attestation systems. Merriv should not invent a competing signature envelope.

merriv mcr predicate runs/release \
  > mcr.predicate.json
cosign attest --yes \
  --type https://github.com/niansia/Merriv/attestations/model-change-report/v0.1 \
  --predicate mcr.predicate.json \
  <artifact-reference>

Cosign supplies the in-toto subject and Statement envelope, then signs it. Merriv only emits the predicate body; it does not hold signing keys or claim that content identity authenticates its producer.

For a registry-native prototype, emit a complete unsigned Statement or an OCI 1.1 subject/referrer layout:

merriv mcr statement runs/release \
  --subject-name registry.example/model:v42 \
  --subject-sha256 <64-hex-digest>
merriv mcr oci-layout runs/release \
  --subject-name registry.example/model:v42 \
  --subject-digest sha256:<manifest-digest> \
  --subject-size <manifest-size> \
  --output runs/release-oci

See supply-chain interoperability for the exact boundary: a Model Change Report does not replace SLSA provenance, SPDX/CycloneDX ML-BOM, OpenSSF Model Signing/Sigstore, OCI transport, or consumer-side deployment policy.

Regulated release records

The immediate design-partner target is teams that already need traceable model release records in healthcare, finance, automotive, critical infrastructure, or other governed environments. This project is not a compliance certification. The EU AI Act high-risk requirements now apply from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I product systems; see the official regulation. The practical product claim is narrower: a Model Change Report can retain exact release evidence that an organization's own risk, quality, and compliance process decides it needs.

Development

uv sync --frozen --extra dev --extra onnx
uv run --frozen ruff check .
uv run --frozen mypy src
uv run --frozen pytest

CI also checks schema drift, content-identity vectors, Rust interoperability, reproducible builds, the composite action, cross-platform package smoke tests, CPU ONNX evidence on Linux and Windows, dependency review, CodeQL, and OpenSSF Scorecard signals. Version 0.1.0a3 is published on PyPI through Trusted Publishing; the release workflow builds, attests, and publishes only a tag that exactly matches the package version.

Merriv is pre-alpha. Public contracts use explicit schema versions, but stability is not promised until v1.0. The project currently has no publicly verified external adopter; repository-owned integrations are not counted as adoption. The public name, Python distribution, import namespace, and CLI are all merriv. See the naming decision.

Contributing and security

Read CONTRIBUTING.md before opening a change. Compatibility reports and questions follow SUPPORT.md. Report vulnerabilities only through the private security advisory form, not a public issue.

Citation

Citation metadata is available in CITATION.cff.

License

Apache-2.0. See LICENSE.