vernier

Fast, auditable evaluation for 2D vision models. Detection, instance and panoptic segmentation, semantic segmentation, keypoints, and LVIS, all in one package with a Rust core, a Python API, and a standalone CLI.

• Bit-exact with pycocotools==2.0.11, panopticapi, lvis-api and boundary-iou-api in strict mode. Every upstream quirk has a documented disposition (quirks survey).
• Drop-in for pycocotools.cocoeval.COCOeval: change one import, or none. Same constructor, same params mutations, same eval / evalImgs / ious / stats read back off the instance (what carries over).
• 4–24× faster than faster-coco-eval and pycocotools at equal CPU budget, measured on real detector output rather than jittered ground truth (benchmarks).
• Built for real pipelines: training-loop evaluation, multi-rank gathering, per-image tables, error decomposition, calibration, scenario slicing.

Install

pip install vernier               # Python ≥ 3.10, abi3 wheels
cargo binstall vernier-cli        # standalone `vernier` binary, no Python needed
cargo add vernier                 # Rust library

Quickstart: the COCO API you already know

vernier.COCOeval has the same constructor, the same evaluate() / accumulate() / summarize() sequence, and the same .stats as pycocotools. It defaults to parity_mode="strict", so the output is bit-identical (see Status & validation for what that is measured against).

from pycocotools.coco import COCO
from vernier import COCOeval  # was: from pycocotools.cocoeval import COCOeval

coco_gt = COCO("instances_val2017.json")
coco_dt = coco_gt.loadRes("detections.json")

E = COCOeval(coco_gt, coco_dt, iouType="bbox")  # "segm" | "keypoints" | "boundary"
E.evaluate()
E.accumulate()
E.summarize()

Code you can't edit (mmdetection, detectron2, ultralytics, …): patch the symbol before anything imports pycocotools.cocoeval.

import vernier

unpatch = vernier.patch_pycocotools()  # pycocotools.cocoeval.COCOeval -> vernier
run_existing_eval()
unpatch()

The patch is explicit, reversible, and never happens on import. A context manager (vernier.adapters.patched_pycocotools) and a one-fixture pytest recipe are in the pycocotools migration guide.

TorchMetrics' MeanAveragePrecision runs under the patch unchanged, including class_metrics=True and extended_summary=True (tests/python/test_compat_torchmetrics.py asserts the patched and unpatched results are equal). Two params fields are the exception and raise rather than diverge quietly: imgIds subsetting, and areaRng (custom area ranges are a native-Evaluator feature, ADR-0040). The migration guide has the field-by-field table.

Recommended: the native API

The shim exists for migration. New code should use the native Evaluator: immutable configuration, typed results, no pycocotools dependency, and access to everything below.

from pathlib import Path
from vernier.instance import Bbox, CocoDataset, Evaluator

gt = CocoDataset.from_json(Path("instances_val2017.json").read_bytes())
dt = Path("detections.json").read_bytes()

evaluator = Evaluator(iou=Bbox(), parity_mode="strict")
summary = evaluator.evaluate(gt, dt, num_threads=8)

print("\n".join(summary.pretty_lines()))  # the familiar 12-line table
ap = summary.stats[0]

The native Evaluator defaults to parity_mode="corrected", which applies the itemized fixes to upstream bugs. Pass "strict" when your numbers must match published pycocotools results.

Inside a training loop, evaluate on a background worker and feed it tensors directly (torch, JAX, CuPy, NumPy via DLPack, zero-copy):

with evaluator.background(gt) as bg:
    for images, targets in val_loader:
        preds = model(images)
        bg.submit([{"image_id": int(t["image_id"]), **p} for t, p in zip(targets, preds)])
    summary = bg.finalize()

Per-image and per-class diagnostics as Polars DataFrames (pip install "vernier[tables]"):

result = evaluator.evaluate(gt, dt, tables="all")
result.per_class

Paradigms

Pick the submodule that matches your model's output. They have different data models and matching rules, so they are separate evaluators rather than one class with a mode switch (why).

Beyond the COCO API

CLI

A static binary for CI gates and robotics replay pipelines. Output is byte-deterministic (sorted keys, no timestamps), so artifacts diff cleanly.

vernier eval --gt gt.json --dt dt.json --iou-type bbox                  # pycocotools-identical stdout
vernier eval --gt gt.json --dt dt.json --iou-type segm --emit json=result.json --threads 8

Exit codes: 0 success, 1 evaluation error, 2 invalid arguments. Full reference: crates/vernier-cli.

Status & validation

Every row is checked by a parity harness that runs the reference implementation and vernier on the same inputs. "Bit-exact" means parity_mode="strict", against the reference as published on PyPI — vernier's own output is identical on x86-64 and ARM. (A pycocotools you recompile from source can round bbox IoU 1-2 ULP differently on ARM; see ADR-0056 and the migration guide.)

Parity model: ADR-0002. Library-by-library comparison: docs/comparison.md.

Performance

Median wall time at a one-CPU budget (CPU/wall is 1.00 in every cell). COCO val2017 throughout, except the LVIS row, which is LVIS v1 val. Speedup is the other library's time divided by vernier's.

The DT source column is load-bearing. A jittered workload perturbs the ground truth to synthesise detections, which inherits the GT's per-image class distribution — not the distribution a detector actually produces. The bbox and segm rows are measured on real RF-DETR predictions on val2017 (513,808 and 430,516 detections) and the keypoints row on real ViTPose output; the remaining rows still run against synthetic DT, and are labelled as such rather than presented as equivalent.

¹ boundary-iou-api · ² panopticapi · ³ mmsegmentation · ⁴ lvis-api, with 14.8× lower peak memory (1.02 vs 15.08 GiB)

A second real detector on the same kernel and GT gives a different point estimate: RF-DETR Seg-Nano's bbox output (16% fewer detections, higher AP) measures 16.7× / 4.2× / 2.0×. Treat a single-workload speedup as a point estimate, not a property of the library — the release snapshot gives per-stage breakdowns, including the one cell where hotcoco's evaluate kernel is faster than vernier's.

Peak RSS on the real bbox cell is 390 MiB, against 997 MiB (hotcoco), 1384 MiB (faster-coco-eval) and 1429 MiB (pycocotools) — a 2.6–3.7× advantage. Every implementation's working set grows with real detection density, but not by the same factor: against hotcoco the gap widens from 1.4× on the synthetic cell to 2.6× here, because hotcoco's per-detection Python-object overhead scales with detection count while vernier's binary ingest does not.

AMD EPYC-Milan KVM VPS (4 cores x 2 threads = 8 logical CPUs), harness mode release (N=10 measurement reps + 2 warmup, randomised impl order, 5% relative-IQR gate), against hotcoco==1.0.1, faster-coco-eval==1.8.0 and pycocotools==2.0.11. Full methodology, every baseline pin and the complete results: docs/benchmarks.md for every timed row except keypoints, the release snapshot for the round's narrative.

Documentation

• Tutorials: first evaluation, training-loop integration
• Migration guides: pycocotools, faster-coco-eval, panopticapi, lvis-api, mmsegmentation
• How-to guides · Reference · Design decisions (ADRs)

Contributing

just lint && just test && just audit

See CONTRIBUTING.md for the ADR workflow, vendoring policy, and code style.

License

Licensed under either of Apache-2.0 or MIT at your option. Test-only reference implementations used for parity checks are listed in THIRD_PARTY_NOTICES.md. They are not shipped in wheels or binaries.