Metria

Verify inference changes with reproducible evidence.

Metria is an open-source experiment and evidence layer for LLM inference systems. It helps researchers answer a question that raw benchmark numbers do not:

For this model, runtime, workload, hardware, and treatment, what did we ask for, what actually ran, what changed, and are the resulting measurements valid to compare?

Metria sits above inference runtimes and optimization libraries. It is not a serving engine, scheduler, kernel library, or universal quantizer. Systems such as vLLM, llama.cpp, SGLang, MLX, TensorRT-LLM, torchao, LLM Compressor, and custom research code remain responsible for execution; Metria provides the study, provenance, measurement, and comparison layer around them.

Status: 0.1.0 is the first Alpha release. Public APIs remain provisional. Release files are prepared; PyPI publication is a separate maintainer step.

The first complete CLI workflow verifies a local llama.cpp CPU thread-count change using a pinned model and qualified capture provider. It checks what ran, compares sampled token trajectories, and saves a report with both run records. See the 0.1.0 release notes for the supported scope and limits.

Why Metria

Inference research is easy to benchmark and surprisingly hard to compare well. A result can look reproducible while hiding differences in model revisions, tokenizers, runtime settings, cache formats, hardware, measurement methods, or what the runtime actually applied.

Metria is being built around four principles:

1. Requested is not observed. Asking for FP8 KV cache is not proof that the runtime used FP8 KV cache.
2. Comparability is study-specific. A runtime or hardware change may be the variable under study rather than something that makes two runs globally incompatible.
3. Measurements carry method identity. Values with different methods or versions are not silently treated as the same metric.
4. Failed and partial runs are evidence too. Missing observation, timeout, unsupported configuration, or failed cleanup should not be converted into an apparently successful result.

What works today

The standalone KV Fidelity package also supports llama.cpp, MLX, vLLM, and SGLang for its focused KV-cache evaluation workflow. Those component backends should not be confused with the smaller set of first-party runtime adapters already exposed by the Metria core.

Quick start

1. Install Metria

Use Python 3.10–3.14. Once the maintainer publishes the release:

python -m pip install metria==0.1.0
metria --version
metria verify --help

Before publication, install the prepared wheel from the GitHub release, or install from source:

git clone https://github.com/dipeshbabu/metria.git
cd metria
python -m pip install .

metria --version
metria --help

The root package deliberately does not install vLLM, llama.cpp, or other inference engines. Runtime stacks remain optional and user-managed.

2. Verify a local CPU thread change

Follow the local verification guide to build the pinned capture provider and prepare the small example model. Then run:

python tools/qualification/prepare_cpu_verification.py \
  --bin-dir /path/to/qualified/build/bin \
  --model /path/to/stories260K.gguf \
  --output study.json
metria verify study.json --output verification

The output contains manifest.json, report.md, reference.run.json, and candidate.run.json. The command distinguishes completed comparisons, insufficient evidence, invalid comparisons, and execution failures. It reports behavioral observations; it does not apply a universal quality threshold.

Supporting recipe and comparison tools

Define a study recipe

Metria recipes describe requested experiment intent as versioned data. For example:

{
  "schema": "metria.study_recipe.v1",
  "study": {
    "name": "runtime-comparison",
    "runs": [
      {
        "model": {"id": "example/model"},
        "runtime": {"name": "llamacpp"},
        "scenario": {"name": "decode"},
        "measurements": ["kv_fidelity.decode_time_trajectory"]
      },
      {
        "model": {"id": "example/model"},
        "runtime": {"name": "vllm"},
        "scenario": {"name": "decode"},
        "measurements": ["kv_fidelity.decode_time_trajectory"]
      }
    ],
    "comparison": {
      "vary": ["runtime"],
      "control": ["model", "scenario", "measurements"],
      "analyses": ["kv_fidelity.trajectory_match"]
    }
  },
  "measurement_configs": {
    "kv_fidelity.decode_time_trajectory": {
      "prompts": [
        {"id": "p1", "prompt": "The capital of France is"}
      ]
    }
  },
  "environment": {}
}

Validate and fingerprint it:

metria recipe validate study.json
metria recipe digest study.json

Normalize a validated recipe to deterministic JSON with:

metria recipe normalize study.json --output normalized-study.json

normalize reproduces the complete recipe, including prompt text or other sensitive input. Do not treat normalized private recipes as safe-to-publish artifacts.

Inspect data-only geometry/capability and local hardware evidence before a run:

metria inspect study.json
metria inspect study.json --json

Inspection is conservative: it does not infer model geometry from a model name, and an accelerator is not claimed present merely because an environment variable mentions it. See the capability inspection guide.

Persist and compare run evidence

The Python execution APIs return RunRecord values. Persist them with the versioned record API:

from metria import dump_run_record

dump_run_record("run-0001.json", record)

Then compare saved records under the recipe's explicit ComparisonPlan:

metria compare run-0001.json run-0002.json --recipe study.json
metria compare run-0001.json run-0002.json --recipe study.json --json

Use metria verify for the qualified local CPU-thread workflow. General study execution remains available through the Python APIs; the CLI does not expose a generic metria run command.

See the CLI guide and run-record guide.

Core model

Metria does not model an experiment as a fixed list of six peer objects. The study decides what varies and what must stay fixed:

Study = factors + controls + comparison plan

Run = system under test
    × scenario / workload
    × measurement protocol
    @ observed environment

Every run then separates:

requested
    ↓
resolved
    ↓
observed
    ↓
evidence + metrics + artifacts + lifecycle events

• Requested — what the user or recipe asked for.
• Resolved — exact settings, revisions, artifacts, and choices selected before launch.
• Observed — what the runtime and environment report actually ran.

That distinction is central to Metria. A configuration request is experiment intent; observed state is evidence.

See Metria core architecture for the full provisional contract.

Comparison semantics

A ComparisonPlan declares the role of experiment dimensions:

vary      dimensions intentionally changed
control   dimensions that must match
block_by  dimensions used to form comparable groups

For example, a study may intentionally vary runtime and KV-cache treatment, control model/workload/measurement method, and block by hardware class.

Metria therefore does not use one universal "same fingerprint = comparable" rule. Missing controlled evidence is not equality, and methodologically different metrics require an explicit analysis that defines how they may be combined.

Saved-record comparison deliberately requires the study recipe that supplies the comparison plan. Record/evidence digests identify serialized evidence; they do not replace study semantics.

Versioned run evidence

metria.run_record.v1 stores one executed run as strict JSON while preserving:

• requested RunSpec using the same schema as study recipes;
• resolved and observed runtime state;
• lifecycle status;
• metric identity, raw samples, aggregation, uncertainty, and coverage;
• measurement evidence and artifact references;
• lifecycle events and execution provenance.

run_record_digest() covers the full record, including requested intent and local run identity. run_evidence_digest() covers produced evidence while excluding study/run IDs and requested intent. Neither digest is a universal comparability proof.

See run records and comparison.

Runtime adapters

The Metria core currently includes two first-party runtime adapters:

llama.cpp

The current adapter supports local llama.cpp command-line execution, GGUF model paths, explicit runtime settings, KV-cache treatments, binary identity hashing, requested/resolved/observed evidence, and optional decode-time token-ID capture with a compatible patched binary.

See the llama.cpp runtime guide.

vLLM

The current adapter uses the offline vllm.LLM API, keeps vLLM as an optional lazy dependency, supports native token-ID capture, and separates configured runtime state from introspected applied engine state.

See the vLLM runtime guide.

Runtime support is intentionally narrow while the common adapter contract is being hardened. Metria should prefer adapters over reimplementing upstream runtimes.

Measurement and fidelity

The first Metria measurement bridge is decode-time token trajectory capture. Each run retains its own token-ID trajectory evidence and prompt fingerprints; the trajectory agreement score is derived only when a valid reference/candidate pair is compared.

This keeps two concepts separate:

run-local evidence != pairwise fidelity metric

The pairwise trajectory analysis is compatible with the current KV Fidelity trajectory methodology while avoiding KV Fidelity's legacy module-global backend dispatch.

See the trajectory measurement guide.

Focused components

Metria remains a monorepo with focused components that keep their own package identities and release lifecycles.

KV Fidelity

Reference-anchored behavioral evaluation for KV-cache compression and runtime changes.

python -m pip install "./components/kv-fidelity"
kv-fidelity --help

See KV Fidelity and its quick start.

TurboQuant Reference

Portable NumPy/SciPy reference implementations of PolarQuant, QJL, and TurboQuant KV-cache compression.

python -m pip install "./components/turboquant-reference"
python components/turboquant-reference/benchmarks/examples/demo.py

See TurboQuant Reference.

What Metria is not

Metria is not intended to become:

• another inference server;
• a replacement for vLLM, llama.cpp, SGLang, MLX, or TensorRT-LLM;
• a reimplementation of every quantization algorithm;
• a universal scalar fidelity score;
• an automatic recommendation engine before measurement uncertainty and failure semantics are mature;
• a single environment containing every inference runtime;
• a training-optimization framework.

Roadmap

The current design roadmap is tracked in issue #50.

Near-term work is focused on:

1. Observed runtime identity — stronger served model/tokenizer/applied-config evidence.
2. Verifier coverage — extend the local CPU-thread workflow to additional qualified inference changes, with trustworthy performance and acceptance policies.
3. Runtime qualification — exercise the shared contract against first-party adapters and add hardware-qualified evidence lanes.
4. Artifact provenance — immutable model/data verification and manifest identity.
5. Shared systems APIs — move benchmark, timeout, diagnostics, and reusable KV Fidelity logic behind Metria protocols rather than adding more standalone scripts.

Later phases can add broader evaluation suites, more runtime/optimization adapters, experiment matrices, Pareto visualization, and constrained search. Automatic recommendation should come only after the evidence layer is mature enough to support it.

Repository layout

src/metria/                 Shared Metria core
metria_tests/               Core contract tests
components/
  kv-fidelity/              Focused fidelity evaluator
  turboquant-reference/     Portable algorithm reference
docs/                       Current guidance and architecture
research/                   Dated studies and investigations
artifacts/                  Retained experiment evidence
tools/                      Existing diagnostics and benchmark utilities

Current guidance belongs in docs/. Dated research conclusions and negative results belong in research/. Generated evidence belongs in artifacts/. Historical evidence should remain historical rather than being silently rewritten to match newer conclusions.

Development

git clone https://github.com/dipeshbabu/metria.git
cd metria

uv sync --all-packages
uv run pre-commit install
uv run pytest

Core-only checks:

uv run pytest metria_tests -v
uv run mypy src/metria
uv run ruff check src/metria metria_tests
uv run ruff format --check src/metria metria_tests

Build the workspace distributions independently:

uv run python -m build .
uv run python -m build components/kv-fidelity
uv run python -m build components/turboquant-reference

Backend-specific inference dependencies remain optional. Install only the stack needed for the runtime under test.

Contributing

See CONTRIBUTING.md, GOVERNANCE.md, and SUPPORT.md.

Changes to experiment semantics, metric methodology, comparison rules, packaging, or release policy should be discussed before those contracts are stabilized. New runtime, evaluator, benchmark, or optimization work should plug into the common Metria contracts rather than creating a parallel architecture.

Citation

Use CITATION.cff when Metria supports your work. When relying on a specific result under research/, cite that report as well so readers can recover its model, runtime, hardware, configuration, and date.

License

Apache License 2.0. See LICENSE and NOTICE.