benchprobe

Install: pip install benchprobe

A Python psychometrics library for AI benchmark scores. Give it a model × benchmark matrix and it answers four questions the evaluation literature keeps asking by hand:

What makes it different from writing these yourself: every estimator is validated by reproducing the published numbers of three studies from hash-pinned data — 44 acceptance tests, tolerances declared before the code was written and never loosened, a reproducibility ledger stating for each number whether it is recomputed, statistically reproduced or regenerated, and a review by a model from a different family on record. The studies are the library's validation suite, not its purpose; your data is.

NumPy, SciPy, pandas, scikit-learn, factor_analyzer. No GPU, no service, no UI. Python ≥ 3.12.

Install

pip install benchprobe            # from PyPI, once v0.1 is tagged
pip install git+https://github.com/louisyzhu/benchprobe   # or from the repository

For development: uv sync && uv run pytest -m smoke (locked environment; the smoke set runs in about 30 s).

Ten lines on your own data

import pandas as pd
import benchprobe.irt as irt
import benchprobe.measure as measure
import benchprobe.predict as predict
from benchprobe.scores import ScoreMatrix

long = pd.read_csv("scores.csv")  # columns: model, item, score (0–1), release_date
sm = ScoreMatrix.from_long(long, release_date="release_date")

z = sm.complete()  # rows scored on every benchmark
z = (z - z.mean()) / z.std(ddof=0)
print(measure.kmo(z), measure.parallel_analysis(z).k_retained)
print(measure.first_factor_share(z, k=3))  # share of common variance on the first of k factors
print(measure.date_r2(measure.efa(z, k=1).scores, sm.grid().days, form="logistic"))

res = predict.lobo(sm.grid(), targets=["your_criterion"], k=3, learners=["ridge"])
print(res.metrics[["target", "rung", "learner", "test_rmse", "test_r2"]])

fit = irt.fit_single_stage(irt.panel_from_long(sm.long()))
print(
    irt.ability_table(fit, panel=irt.panel_from_long(sm.long()))[["model_id", "theta", "se_theta"]]
)

examples/quickstart.py is that script on simulated data, end to end, in about a second: uv run python examples/quickstart.py.

Conventions: scores are proportions in [0, 1] (scale="percent" divides by 100); a missing cell is NaN wide and an absent row long; a repeated (model, item) cell is an error unless you say what it means (aggregate="mean"). ScoreMatrix does bookkeeping only — every number comes from the four layers.

The three studies it reproduces

benchprobe.studies holds one function per study; nothing outside that package knows a benchmark by name.

docs/reproducibility.md is the ledger: one row per number, with archive value, recomputed value, tolerance and tier. docs/decisions.md records every design decision and every finding about the archives — including the ones that are corrections to the archives themselves. Full acceptance run: uv run pytest -m golden (46 min; the One Capability ladder is most of it).

What it is not

Not a leaderboard, a scraper, a model runner, a dashboard or an "audit in an afternoon". It computes statistics on tables you already have, and it carries no claim its validation studies do not carry. If you need a general benchmark-auditing product, see tinyBenchmarks, MetaBench or PSN-IRT; if you need the specific estimators those papers use, validated against their numbers, this is it.

Rules the repository runs by

CLAUDE.md — ten rules, checked by a smoke test. The ones a user should know: a golden number, tolerance or seed is never changed to make a test pass (a failing golden test is a finding and is reported); every reported number has a test; every output states its reproducibility tier; the environment is pinned; nothing is published by an agent.

Data

Three snapshots are vendored under benchprobe/data/, each with a MANIFEST.json whose entries load_snapshot verifies on every load and whose own hash the golden tests pin: the One Capability analysis table (frontier-ai-economic-validity, CC BY 4.0; figures originate with Artificial Analysis and Epoch AI), the JUDGe item bank (llm-judge-reliability, CC BY 4.0), and the Price of Intelligence panel with three Phase-2 output files (Zenodo, hash-identical to the archive's own manifest). BENCHPROBE_DATA_DIR points at alternative snapshot folders. Nothing here performs inference or needs credentials.

Licence and citation

MIT for the code; the vendored data keeps its CC BY 4.0 terms. Cite the release (CITATION.cff) and the study whose numbers you rely on.