hotcoco

hotcoco is a perception evaluation toolkit, written in Rust with Python bindings. It's a drop-in replacement for pycocotools — the same numbers bit for bit, up to 113× faster — plus the analysis that usually lives in separate tools: TIDE error analysis, confusion matrices, confidence calibration, model comparison, label-error detection, and a dataset browser.

It covers detection — boxes, masks, keypoints, and oriented boxes on the COCO, LVIS, and Open Images protocols — panoptic segmentation, verified against panopticapi, and semantic segmentation, verified against mmsegmentation. Tracking is planned, on the same engine.

Pure Rust, available as a Python package, CLI tool, and Rust library.

Documentation | Changelog | Roadmap

Performance

Bbox evaluation on COCO val2017 runs in 0.05s against 5.11s for pycocotools; segm is ~111× faster and keypoints ~55×. Peak memory is a third of pycocotools' or less at every scale measured. Two other Rust evaluators, ultrafast-pycocotools and vernier, are in the same speed class; the benchmarks page compares all five on time and memory. The precision, recall, and scores arrays and every summary metric are bit-identical to pycocotools', so your AP scores don't change.

Full tables, hardware, the Objects365 scale run, phase breakdowns, and parity verification: Benchmarks.

Get started

pip install hotcoco

No Cython, no C compiler, no Microsoft Build Tools. Prebuilt wheels for Linux, macOS, and Windows.

Already using pycocotools? One line:

from hotcoco import init_as_pycocotools
init_as_pycocotools()

Or use it directly — the API is identical:

from hotcoco import COCO, COCOeval

coco_gt = COCO("instances_val2017.json")
coco_dt = coco_gt.load_res("detections.json")

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.run()

What's included

Evaluate

• COCO, LVIS, and Open Images evaluation — bbox, segmentation, keypoints, and oriented bounding box (OBB); all standard metrics plus LVIS federated eval (APr/APc/APf) and Open Images hierarchy-aware eval (group-of matching, GT expansion). OBB evaluation uses rotated IoU via polygon clipping for aerial imagery, document analysis, and scene text. See the evaluation guide and LVIS and Open Images.
• Panoptic quality — hotcoco.panoptic.PanopticEval computes PQ, SQ, and RQ for all, things, and stuff from COCO panoptic JSON + PNG files, or from annotations carrying RLE or polygon masks with no PNG files; pq_compute is a drop-in for panopticapi's. The numbers match panopticapi on COCO val2017. See the panoptic guide.
• Semantic segmentation — hotcoco.semantic.SemanticEval computes mIoU, fwIoU, mAcc, and aAcc from class-id label maps, fed one batch at a time from a validation loop or read from folders of PNG files; the state is one confusion matrix, so shards merge and the evaluator pickles. The numbers match mmsegmentation and torchmetrics on frozen label maps. See the semantic guide.
• Streaming evaluation — StreamingEval matches each image as its detections arrive, so a validation epoch ends with only accumulate() and summarize() left to run; the numbers are identical to a batch run. It takes the detector's prediction array directly, and shards streamed on separate ranks merge into one result. See streaming evaluation.
• Evaluation reports — ev.report() returns metrics, per-class and per-group breakdowns, plottable PR curves, and a provenance field that says whether each number is comparable to a published leaderboard or is a hotcoco extension. See the evaluation report.

Diagnose

• TIDE error analysis — classifies every false positive and false negative into one of six error types and reports how much AP each type costs. See TIDE errors.
• Confusion matrix — cross-category matching with per-class breakdowns. See confusion matrix.
• Confidence calibration — ECE/MCE metrics and reliability diagrams measure whether your model's confidence scores are meaningful. See calibration.
• Per-image diagnostics and label errors — per-image F1/AP scores, automatic detection of wrong labels and missing annotations in your ground truth. See diagnostics.
• Model comparison — hotcoco.compare(eval_a, eval_b) with per-metric deltas, per-category AP breakdown, and bootstrap confidence intervals for statistical significance. See model comparison.
• F-scores — F-beta averaging over precision/recall curves, analogous to mAP. See F-scores.
• Sliced evaluation — re-accumulate metrics for named image subsets (indoor/outdoor, day/night) without recomputing IoU. See sliced evaluation.
• Plotting — publication-quality PR curves, per-category AP, confusion matrices, and TIDE error breakdowns. Light and dark themes (cyanotype, cyanotype-dark) with paper_mode for LaTeX/PowerPoint embedding. report() generates a single-page PDF summary. pip install hotcoco[plot]. See plotting.

Explore your data

• Dataset browser — coco.browse() / coco explore opens a local browser with category filter, annotation overlays (bbox/segm/keypoints/OBB), hover-to-highlight, zoom/pan, and detection comparison. Pass eval= to enable an interactive eval dashboard with PR curves, confusion matrix, TIDE errors, calibration, and per-image F1. pip install hotcoco[browse]. See Dataset browser.
• Dataset healthcheck — 4-layer validation (structural, quality, distribution, GT/DT compatibility) catches duplicate IDs, degenerate bboxes, category imbalance, and more. See healthcheck.
• Format conversion — COCO ↔ YOLO, Pascal VOC, CVAT, DOTA (oriented boxes), and Open Images CSV, from Python or the CLI. See format conversion.

Compose and integrate

• Metric functions on plain arrays — hotcoco.metrics and hotcoco.primitives expose the engine as free functions, the way sklearn.metrics and torchmetrics.functional do. No evaluator, no dataset, no COCO JSON: metrics.average_precision(scores, matched, num_gt=...), primitives.lsap(cost). COCOeval calls the same functions, so the numbers cannot diverge. See metrics and primitives.
• PyTorch integrations — CocoDetection and CocoEvaluator drop-in replacements for torchvision's detection classes; no torchvision or pycocotools dependency required. See PyTorch integration.
• Experiment tracker integration — get_results(prefix="val/bbox", per_class=True) returns a flat dict ready for W&B, MLflow, or any logger. See logging metrics.
• Python CLI (coco) — included with pip install hotcoco; eval, panoptic eval, semantic eval, healthcheck, stats, filter, merge, split, sample, convert, compare, and explore subcommands. See CLI reference.
• Rust CLI (coco-eval) — lightweight eval-only binary with eval, panoptic, and semantic subcommands; cargo install hotcoco-cli. See CLI reference.
• Type stubs — ships with .pyi stubs and py.typed marker for full autocomplete and type checking in VS Code, PyCharm, and other IDEs.
• Rust library — use hotcoco directly in your Rust projects via cargo add hotcoco. See Rust API.

Contributing

Contributions are welcome. See CONTRIBUTING.md for the architecture overview, build and test workflow, and pre-commit checks.

Parity with pycocotools is a hard requirement for any change to evaluation logic; CONTRIBUTING.md describes how to verify it.

License

MIT