vernier

Fast, parity-preserving evaluation for object detection, instance / panoptic / semantic segmentation, boundary IoU, OKS keypoints, LVIS federated, LRP / oLRP error decomposition, and detection-family calibration (ECE / MCE / reliability). Rust core, Python frontend, optional CLI.

60-second example

Post training, if your predictions are already serialized to JSON (CI gate, post-training inspection):

from pathlib import Path
from vernier.instance import Bbox, CocoDataset, Evaluator

gt_bytes = Path("instances_val2017.json").read_bytes()
dt_bytes = Path("detections.json").read_bytes()

dataset = CocoDataset.from_json(gt_bytes)
summary = Evaluator(iou=Bbox()).evaluate(dataset, dt_bytes)
for line in summary.pretty_lines():
    print(line)

In a training loop, vernier supports overlapping eval with the data loading and inference. The matching kernel runs on a worker thread, so submit(...) returns immediately and the main thread keeps moving. Passing a CocoDataset reuses the parsed-once GT and its per-kernel derivation cache across every epoch (ADR-0020). On a dedicated validation pass (no trainer competing for cores), pass num_threads=N to parallelise the matching kernel inside the worker (ADR-0047):

from pathlib import Path
from vernier.instance import Bbox, CocoDataset, Evaluator

gt = CocoDataset.from_json(Path("instances_val2017.json").read_bytes())
evaluator = Evaluator(iou=Bbox())
with evaluator.background(gt, num_threads=8) as bg:  # default: single core
    for images, targets in val_loader:
        # torchvision detection API shape: list[dict] of length batch_size,
        # each with "boxes" (N,4 xywh), "scores" (N,), "labels" (N,) as
        # torch.Tensor. vernier consumes any DLPack-producing array library
        # (torch, jax, cupy, numpy) zero-copy.
        predictions = model(images)
        bg.submit([
            {"image_id": int(t["image_id"]), **p}
            for t, p in zip(targets, predictions)
        ])
    summary = bg.finalize()
print("AP =", summary.stats[0])

Both end in the same 12-line pycocotools-shaped Summary; docs/tutorials/first-evaluation.md walks each end-to-end.

Benchmarks

Thread scaling (num_threads, each cell pinned to that many CPUs — nt=8 is every library's out-of-the-box configuration on this 8-vCPU host):

faster-coco-eval ≥1.8 and hotcoco are multi-threaded too, so every column compares equal CPU budgets. Its parallelism pays off on boundary IoU but is flat on segm and keypoints, which is why vernier's lead widens with cores there.

^† The Objects365 row is harness mode dev (one measurement rep per impl, no IQR gate) — pycocotools alone needs ~6 minutes per rep at that size. Every other row is release mode.

Median total-stage wall time on a KVM VPS (AMD EPYC-Milan, 4 cores × 2 threads = 8 logical CPUs, x86_64 — not a bare-metal Milan box), harness mode release (N=10 measurement reps + 2 warmup, randomised impl order, 5% relative-IQR gate per impl), build profile = cargo release defaults (opt-level=3, lto=thin, codegen-units=1, no target-cpu) — same as the PyPI wheel. Every impl gets one CPU: faster-coco-eval ≥1.8, hotcoco and mmsegmentation are multi-threaded by default, so each runner is pinned to the cell's CPU budget and each result records its CPU/wall ratio as evidence (ADR-0049). vernier scales with num_threads — the thread-scaling tables on the benchmarks page cover 1/2/4/8. Full per-cell breakdown (including IQRs), peak and eval-delta memory, and methodology in docs/benchmarks.md; per-library comparison of when to pick which in docs/comparison.md.

Baselines pinned for these numbers — pycocotools==2.0.11, hotcoco==1.0.1, faster-coco-eval==1.8.0, panopticapi @ 7bb4655, boundary-iou-api @ 37d2558, mmsegmentation @ c685fe6 (vendored), lvis-api @ 031ac21 (PyPI lvis==0.5.3). All cells were measured at HEAD e361050ef582 (machine fingerprint 59aab88b17f4). That is a different host from the 2026-05 snapshot's 37652a58e939, and the CPU budget above changed what a "single-thread" cell means, so absolute numbers are not comparable with earlier snapshots — ratios within this one are. Each baseline is locked in its own uv-managed venv per ADR-0017.

Install

pip install vernier                  # Python wheel
cargo add vernier                    # Rust library (all paradigms)
cargo install vernier-cli            # `vernier` CLI binary

Wheels ship for linux x86_64 / aarch64 (glibc + musl), macOS x86_64 / arm64, and windows x64.

On the Rust side, vernier is a facade (ADR-0048): it re-exports the paradigm crates under one dependency and one module map — vernier::{instance, mask, panoptic, semantic, partial} — mirroring the Python namespace. It holds no code of its own, so depending on a leaf crate directly (cargo add vernier-core for bbox / segm / boundary / keypoints AP alone) is equally supported; the three optional paradigms can also be trimmed in place with default-features = false. Note the asymmetry: cargo add vernier gets the library, cargo install vernier-cli gets the binary — the CLI stays out of the library's dependency tree so clap never lands in a consumer's build.

Status & validation

Pre-1.0; public API is unstable. See docs/adr/ for the design decisions shaping it.

pycocotools==2.0.11 is the de-facto reference for COCO evaluation — slow, unmaintained, and full of edge-case quirks. Faster reimplementations exist, but each silently fixes some quirks and not others, so you discover the divergences empirically. vernier takes a third path:

• Auditable parity. Every divergence from pycocotools is filed in the quirks survey under ADR-0002 as either strict (bit-equal output, even when vernier's implementation is structurally different) or corrected (opt-in opinionated fix). Strict is the default; corrected fixes are itemized so you always know when your numbers diverge from a reference run. A drop-in shim (vernier.patch_pycocotools()) keeps existing pycocotools-based scripts working with one line.
• Rust core, Python frontend. The matching kernel is pure Rust with runtime SIMD dispatch; the FFI layer is data conversion only. The CLI ships as a static binary, so CI pipelines call vernier without provisioning a Python interpreter.
• One toolkit instead of five. bbox / segm / boundary / keypoints AP, panoptic PQ, semantic mIoU, LVIS federated, oLRP error decomposition, and detection-family calibration all live behind one Python API and one CLI — folded over a single matching pass. Per-paradigm migration guides under docs/migrate/ show how to replace pycocotools, faster-coco-eval, panopticapi, lvis-api, and mmsegmentation one at a time.
• Scenario slicing + cross-run aggregation. A partition manifest (weather, time_of_day, …) feeds vernier eval --manifest for per-slice headline metrics and vernier aggregate for cross-run corruption tables (mPC / rPC) — one matching pass, N slices (ADR-0046).

Per-paradigm parity status:

Three-tier parity model: ADR-0002; per-library comparison: docs/comparison.md.

Three evaluation paradigms

They have different data models, different matching rules, and different parity oracles:

• vernier.instance — detections with scores → bbox / segm / boundary / keypoints AP.
• vernier.panoptic — RGB-encoded panoptic PNGs + segments_info JSON → PQ.
• vernier.semantic — single-channel class-id label maps → mIoU / FWIoU / pAcc / mAcc.

See Three paradigms.

Documentation

• Tutorials — docs/tutorials/
• Migration guides (from pycocotools, faster-coco-eval, panopticapi, lvis-api, mmsegmentation) — docs/migrate/
• How-to — docs/how-to/
• Reference — docs/reference/
• Design / ADRs — docs/adr/
• Comparison vs pycocotools / faster-coco-eval / panopticapi / boundary-iou-api / lvis-api / mmsegmentation — docs/comparison.md

Contributing

Local checks: just lint && just test && just audit. The full contributor workflow (ADR lifecycle, vendoring policy, code style) is in CONTRIBUTING.md. Repository layout and common just recipes are in CLAUDE.md.

License

Dual-licensed under Apache-2.0 or MIT at your option.

Third-party code

vernier vendors a small number of test-only reference implementations to support parity testing. None of this code is included in published wheels or linked into the Rust binary. See THIRD_PARTY_NOTICES.md for the full inventory and license attributions.