Metria

Metria verifies changes to LLM inference systems.

Give it a reference configuration and a candidate configuration. Metria checks what actually ran, rejects an unjustified comparison, measures behavioral change, and evaluates the result against your explicit acceptance criteria.

Test inference changes before you ship them.

The current end-to-end CLI verifies a local llama.cpp CPU thread-count change. It holds the pinned model, qualified capture provider, prompt workload, and other settings fixed, checks native readback, and retains both runs with a reviewable report. The Python runtime adapters and research components support additional building blocks; their presence does not establish a qualified CLI workflow.

Start with one change

For the current development features, install a checkout with Python 3.10–3.14:

git clone https://github.com/dipeshbabu/metria.git
cd metria
python -m pip install .

Prepare the pinned runtime, model, and reference/candidate recipe using the local verification guide, then run:

metria verify study.json --output verification --json

The root Python package has no runtime dependencies. Native engines and models are installed separately. Each output directory must be new.

Published Metria 0.1.0 provides the initial local verifier. The development checkout adds the canonical result, richer diagnostics, explicit policies, performance deltas, and CI integration described here. See the release notes for the published release's scope.

Start with the copyable workflows: a qualified local configuration change, synthetic PASS/FAIL/invalid-comparison fixtures, and clearly staged templates for runtime upgrades, KV precision, quantization, and build regressions. Synthetic examples require no model or GPU and are labeled as test evidence throughout.

Read the decision

Every result separates four questions:

verification/
  verification.json
  report.md
  reference.run.json
  candidate.run.json

The original manifest.json filename remains a compatibility alias. The canonical JSON and Markdown report include separate lifecycle, comparison, and policy states. Timeouts, failed runs, insufficient evidence, and unexpected differences remain visible instead of becoming a successful comparison.

Metria distinguishes requested → resolved → observed: a requested setting states intent; the resolved artifact/configuration identifies what will launch; native observation checks what actually applied. A requested CPU thread change must appear in runtime readback before its behavioral result is accepted. An undeclared tokenizer or generation change prevents comparison.

The first behavioral method is KV Fidelity-compatible token-trajectory analysis. It reports prefix agreement, exact matches, first-divergence positions, category rates, and prompt identifiers that help locate drift without dumping prompt text. See behavioral diagnostics.

The current systems method reports compatible cold-process request latency, including startup and model loading. TTFT, decode-only throughput, and device/KV memory remain unavailable until an authoritative method is supported. The report retains trial policy and limitations; see performance evidence.

VERIFIED means valid comparison and completed analysis. Add an explicit acceptance policy for PASS/FAIL. Metria supplies no universal behavior or safety threshold; PASS means your stated criteria were met after the evidence gates passed.

Use it in code review and CI

Run the same command as a job step and preserve its exit status:

- name: Verify inference change
  run: metria verify .metria/change.json --output metria-verification --json

The CI guide defines distinct failure exits, retained artifacts, and a GitHub Actions summary adapter. The copyable workflow keeps the report and records even when verification fails. A real CI workload needs the same qualified runtime/model provisioning as local use.

Scope and qualification

See runtime qualification. A mocked contract test, installed engine, or visible GPU is not a real-engine qualification. New routes must prove their identity, requested change, and required captures.

Metria focuses on one inference change at a time. Broad experiment matrices, generic benchmark orchestration, arbitrary plugin discovery, automatic search, Pareto dashboards, additional runtimes solely for breadth, training optimization, and hosted services are deferred until they directly strengthen this verifier. The product decision is tracked in #50.

Supporting APIs and repository

Use the recipe CLI, inspection guide, and saved-record comparison to prepare or debug a verification. The core architecture documents the reusable study/run/evidence machinery behind that workflow.

src/metria/ owns verification and shared evidence contracts. components/kv-fidelity/ owns focused behavioral methods; components/turboquant-reference/ owns the research reference implementation. Current guidance lives in docs/; dated investigations in research/; retained evidence in artifacts/. Historical results keep their original scope and known gaps under the provenance policy.

For development:

uv sync --locked --all-packages
uv run pre-commit install
uv run python -m pytest

See Contributing, Governance, and Support. Cite CITATION.cff and the specific retained report when relying on an experimental result.

Original software uses Apache-2.0; see LICENSE and NOTICE. Third-party/model-derived material retains its separately documented rights.