vernier

Fast, auditable evaluation for 2D vision models. Detection, instance and panoptic segmentation, semantic segmentation, keypoints, and LVIS, all in one package with a Rust core, a Python API, and a standalone CLI.

• Bit-exact with pycocotools==2.0.11, panopticapi, lvis-api and boundary-iou-api in strict mode. Every upstream quirk has a documented disposition (quirks survey).
• Drop-in for pycocotools.cocoeval.COCOeval: change one import, or none. Same constructor, same params mutations, same eval / evalImgs / ious / stats read back off the instance (what carries over).
• 3.7–19× faster than faster-coco-eval and pycocotools at equal CPU budget (benchmarks).
• Built for real pipelines: training-loop evaluation, multi-rank gathering, per-image tables, error decomposition, calibration, scenario slicing.

Install

pip install vernier               # Python ≥ 3.10, abi3 wheels
cargo binstall vernier-cli        # standalone `vernier` binary, no Python needed
cargo add vernier                 # Rust library

Quickstart: the COCO API you already know

vernier.COCOeval has the same constructor, the same evaluate() / accumulate() / summarize() sequence, and the same .stats as pycocotools. It defaults to parity_mode="strict", so the output is bit-identical (see Status & validation for what that is measured against).

from pycocotools.coco import COCO
from vernier import COCOeval  # was: from pycocotools.cocoeval import COCOeval

coco_gt = COCO("instances_val2017.json")
coco_dt = coco_gt.loadRes("detections.json")

E = COCOeval(coco_gt, coco_dt, iouType="bbox")  # "segm" | "keypoints" | "boundary"
E.evaluate()
E.accumulate()
E.summarize()

Code you can't edit (mmdetection, detectron2, ultralytics, …): patch the symbol before anything imports pycocotools.cocoeval.

import vernier

unpatch = vernier.patch_pycocotools()  # pycocotools.cocoeval.COCOeval -> vernier
run_existing_eval()
unpatch()

The patch is explicit, reversible, and never happens on import. A context manager (vernier.adapters.patched_pycocotools) and a one-fixture pytest recipe are in the pycocotools migration guide.

TorchMetrics' MeanAveragePrecision runs under the patch unchanged, including class_metrics=True and extended_summary=True (tests/python/test_compat_torchmetrics.py asserts the patched and unpatched results are equal). Two params fields are the exception and raise rather than diverge quietly: imgIds subsetting, and areaRng (custom area ranges are a native-Evaluator feature, ADR-0040). The migration guide has the field-by-field table.

Recommended: the native API

The shim exists for migration. New code should use the native Evaluator: immutable configuration, typed results, no pycocotools dependency, and access to everything below.

from pathlib import Path
from vernier.instance import Bbox, CocoDataset, Evaluator

gt = CocoDataset.from_json(Path("instances_val2017.json").read_bytes())
dt = Path("detections.json").read_bytes()

evaluator = Evaluator(iou=Bbox(), parity_mode="strict")
summary = evaluator.evaluate(gt, dt, num_threads=8)

print("\n".join(summary.pretty_lines()))  # the familiar 12-line table
ap = summary.stats[0]

The native Evaluator defaults to parity_mode="corrected", which applies the itemized fixes to upstream bugs. Pass "strict" when your numbers must match published pycocotools results.

Inside a training loop, evaluate on a background worker and feed it tensors directly (torch, JAX, CuPy, NumPy via DLPack, zero-copy):

with evaluator.background(gt) as bg:
    for images, targets in val_loader:
        preds = model(images)
        bg.submit([{"image_id": int(t["image_id"]), **p} for t, p in zip(targets, preds)])
    summary = bg.finalize()

Per-image and per-class diagnostics as Polars DataFrames (pip install "vernier[tables]"):

result = evaluator.evaluate(gt, dt, tables="all")
result.per_class

Paradigms

Pick the submodule that matches your model's output. They have different data models and matching rules, so they are separate evaluators rather than one class with a mode switch (why).

Beyond the COCO API

CLI

A static binary for CI gates and robotics replay pipelines. Output is byte-deterministic (sorted keys, no timestamps), so artifacts diff cleanly.

vernier eval --gt gt.json --dt dt.json --iou-type bbox                  # pycocotools-identical stdout
vernier eval --gt gt.json --dt dt.json --iou-type segm --emit json=result.json --threads 8

Exit codes: 0 success, 1 evaluation error, 2 invalid arguments. Full reference: crates/vernier-cli.

Status & validation

Every row is checked by a parity harness that runs the reference implementation and vernier on the same inputs. "Bit-exact" means parity_mode="strict", against the reference as published on PyPI — vernier's own output is identical on x86-64 and ARM. (A pycocotools you recompile from source can round bbox IoU 1-2 ULP differently on ARM; see ADR-0056 and the migration guide.)

Parity model: ADR-0002. Library-by-library comparison: docs/comparison.md.

Performance

Median wall time at a one-CPU budget (CPU/wall is 1.00 in every cell). COCO val2017 throughout, except the LVIS row, which is LVIS v1 val. Speedup is the other library's time divided by vernier's.

¹ boundary-iou-api · ² panopticapi · ³ mmsegmentation · ⁴ lvis-api, with 13.6× lower peak memory (1.11 vs 15.08 GiB)

Peak RSS on the COCO cells is 200 MiB, against 274 MiB (hotcoco), 569 MiB (pycocotools) and 691 MiB (faster-coco-eval).

AMD EPYC-Milan KVM VPS (4 cores x 2 threads = 8 logical CPUs), harness mode release (N=10 measurement reps + 2 warmup, randomised impl order, 5% relative-IQR gate), against hotcoco==1.0.1, faster-coco-eval==1.8.0 and pycocotools==2.0.11. Full methodology, every baseline pin and the complete results: docs/benchmarks.md.

Documentation

• Tutorials: first evaluation, training-loop integration
• Migration guides: pycocotools, faster-coco-eval, panopticapi, lvis-api, mmsegmentation
• How-to guides · Reference · Design decisions (ADRs)

Contributing

just lint && just test && just audit

See CONTRIBUTING.md for the ADR workflow, vendoring policy, and code style.

License

Licensed under either of Apache-2.0 or MIT at your option. Test-only reference implementations used for parity checks are listed in THIRD_PARTY_NOTICES.md. They are not shipped in wheels or binaries.