eval-toolkit

A methodology-aware evaluation harness for binary classification: metrics, bootstrap CIs, calibration, leakage detection, splitting, threshold selection, dataset loading, reproducibility manifests, and a slice-aware orchestrator that ties them together. Pure numpy/scipy/sklearn core; pandas/matplotlib/hypothesis are optional extras; PyTorch / HuggingFace / datasets are consumer-side (never required).

Library-grade by design — every public function is type-annotated, every math kernel is documented with LaTeX + literature references, statistical validity (bootstrap CIs, MDE estimates, paired-difference tests) is built in, and the JSON outputs (results.json / results_full.json / manifest.json) ship with versioned JSON Schemas so downstream parsers can gate on format changes.

Three-tier architecture

┌─ Tier 3 ─ Reproducibility scaffolding ─────────────────┐
│  manifest.json + seeds + git_sha + data_hashes +       │
│  gpu_info + leakage_report (NeurIPS-aligned)           │
├─ Tier 2 ─ Protocol-based orchestration ────────────────┤
│  Scorer / LeakageCheck / Splitter / ThresholdSelector  │
│  DatasetLoader / MetricSpec / MetaLearner / Probe /    │
│  TextTransform / SimilarityStrategy (10 strict)        │
│  SliceAwareScorer / Versioned (outside the 10 strict)  │
├─ Tier 1 ─ Functional core ─────────────────────────────┤
│  pr_auc / roc_auc / ECE variants / Brier / bootstrap_ci│
│  paired_bootstrap_diff / cv_clt_ci / mde_from_ci       │
│  reliability_curve / fit_temperature / fit_isotonic    │
└────────────────────────────────────────────────────────┘

Pick the tier your task needs. Ad-hoc analysis: just call the functional core. Full eval pipelines: implement the Protocols. Every run: capture the manifest.

Documentation

• Getting started — end-to-end walkthrough for new users: install, define a Scorer, build slices, run evaluate(), persist results, add a claim, render a plot.
• Methodology curriculum — 16 chapters on splits, metrics, calibration, evidence gates, prediction artifacts, and more.
• Schema reference — field-by-field semantics for results.v1.json, results_full.v1.json, manifest.v1.json.
• Migration guides — per-version migration hub (v0.7 onward).
• Extending — Protocol-by-Protocol guide for custom Scorers, Splitters, LeakageChecks, ThresholdSelectors, DatasetLoaders, EvidenceGates.
• Repo strategy — how the package is organized, the flat-module layout per ADR 0001, and the v2.0 trigger criteria for any future subpackage split.

Methodology

What good binary-classification evaluation looks like, with each concern mapped to the toolkit primitive that operationalizes it.

• docs/source/methodology/ — the curriculum (16 chapters). Recommended reading order: leakage → splits → thresholds → calibration → comparison → bootstrap → length_stratification → text_dedup → versioning → fairness → reproducibility → testing → reading_list.
• docs/source/MIGRATION.md — per-version migration guides (v0.7 onward; v0.49 / v0.50 / v0.51 included as of v0.51.0).
• docs/source/roadmap.md — forward-looking tracker; v1.0.0 path; consumer gap-doc cross-links.

Extending eval-toolkit

How to plug your own scorers / leakage checks / splitters / loaders / threshold selectors into the harness.

• docs/source/extending.md — Protocol-by-Protocol guide, ~50-line full-harness recipe, project-layout pointer.

Worked examples

• docs/source/examples/ — Sphinx / MyST-NB executable notebooks covering: the evaluation harness, metrics + bootstrap, calibration, claims-and-gates, leakage detection, cross-corpus contamination scanning, character-injection adversarial sweeps, callable-embedder dedup, and the activation-delta probe.

Install

uv venv
uv pip install -e .[dev]

For consumers who only need the math kernels (no plotting, no pandas):

pip install eval-toolkit                        # core only: numpy/scipy/sklearn
pip install "eval-toolkit[plotting]"            # adds matplotlib + pillow
pip install "eval-toolkit[dataframe]"           # adds pandas
pip install "eval-toolkit[all]"                 # everything

Quick examples

Metrics

import numpy as np
from eval_toolkit.metrics import pr_auc, roc_auc, expected_calibration_error

rng = np.random.default_rng(42)
y = rng.integers(0, 2, size=200)
# Clip to [0, 1] — ECE only meaningful on calibrated probabilities.
s = np.clip(y + rng.normal(0, 0.3, size=200), 0, 1)

print(f"PR-AUC: {pr_auc(y, s):.3f}")
print(f"ROC-AUC: {roc_auc(y, s):.3f}")
print(f"ECE (10 bins): {expected_calibration_error(y, s, n_bins=10):.3f}")

Bootstrap confidence intervals

from eval_toolkit import bootstrap_ci, paired_bootstrap_diff
from eval_toolkit.metrics import pr_auc

ci = bootstrap_ci(y, s, pr_auc, n_resamples=1000, rng=42)
print(f"PR-AUC: {ci.point_estimate:.3f}  95% CI: [{ci.ci_low:.3f}, {ci.ci_high:.3f}]")

# Paired bootstrap on the lift between two scorers (s_baseline must be in [0, 1] too).
s_baseline = np.clip(rng.normal(0.5, 0.3, size=200), 0, 1)
diff = paired_bootstrap_diff(y, s_baseline, s, pr_auc, n_resamples=1000, rng=42)
print(f"Δ PR-AUC: {diff.delta:.3f}  overlaps zero: {diff.overlaps_zero}")

Temperature scaling (Guo et al. 2017)

import numpy as np
from eval_toolkit import fit_temperature

rng = np.random.default_rng(42)
logits = rng.normal(size=(500, 2))
labels = (logits[:, 1] > logits[:, 0]).astype(int)
result = fit_temperature(logits, labels)
print(f"Optimal T: {result['temperature']:.3f}")
print(f"NLL: {result['nll_pre']:.3f} -> {result['nll_post']:.3f}")

Reproducibility manifest (NeurIPS-aligned)

import tempfile
from pathlib import Path
from eval_toolkit import make_manifest, write_manifest

with tempfile.TemporaryDirectory() as run_dir:
    # data_files: {name: path} → eval_toolkit hashes the files for you;
    # versioned: any object with a `version` attribute (e.g. a scorer or
    # leakage check) is captured by name → version in the manifest.
    manifest = make_manifest(
        run_id="quickstart-demo",
        config={"threshold_criterion": "max_f1", "seed": 42},
        seeds={"global": 42, "bootstrap": 42},
    )
    write_manifest(manifest, Path(run_dir))
    # → run_dir/manifest.json: schema_version, git_sha, dirty_flag, code_versions,
    #   env (python+platform), seeds, data_hashes, versioned_objects, gpu_info

Modules

Fast iteration loop

For development, skip slow tests with:

make fast              # or: nox -s fast
# under the hood: uv run pytest -m "not slow" -q

CI runs the full suite (including slow) on every push. The slow marker is applied to tests exceeding ~2s (mostly Hypothesis property tests with large max_examples and a few bootstrap tests with n_resamples >= 200). make fast keeps the developer iteration loop under ~30 seconds.

Standards

See STYLE.md for the full reconciled coding standards (formatting, naming, errors, docstrings, tests, packaging).

Versioning

Semver from v0.1.0. See CHANGELOG.md.

License

MIT — see LICENSE.