honestboard

Honest uncertainty for agent leaderboards. Give it the per-task results of every system on a benchmark and it tells you:

• Is this SOTA claim real? An exact paired test of a new record against the record it replaced.
• What can this benchmark resolve? The smallest difference it detects with 80% power, and how many tasks a one-point gap would need.
• How sure is the ranking? Rank intervals and the probability each system is the best.
• Who could still be the best? A 95% confidence set of the best system.
• What is the winner's score, corrected? The top score on a leaderboard is biased upward because it was picked for being highest. honestboard removes most of that selection bias.
• How many record claims survive? An audit of every record step in a leaderboard's history, with a Holm correction.
• Certify records as they arrive. certify_chain runs a sequential rule along the submission history: the exact sign-test p-value conditional on the record event (with a per-date factor for submissions that share a date), at levels set by alpha-spending or by the online fallback rule (family-wise error over the whole chain, any number of records, no independence assumption), by tenure-indexed spending on the certified baseline, or by LORD++ (online FDR; certified records earn back budget), against the leaderboard holder or against the last certified record.

It needs per-task outcomes (which tasks each system solved), not just the headline score. A checkout of the SWE-bench experiments repository loads with one call (from_swebench_experiments); any other leaderboard works once you write a board file (format below).

Quick start

from honestboard import Board

b = Board.from_json("swebench_verified.json")   # board files may carry a "clusters" map (task -> repository)

b.compare("20250928_trae_doubao_seed_code", "20251205_sonar-foundation-agent_claude-opus-4-5")
b.rank_summary(top=10)              # 95% rank intervals and top-rank frequency
b.confidence_set_best()             # systems that cannot be ruled out as best
b.record_audit()                    # every record step: selective exact test (with the per-date factor), Holm-adjusted
b.record_lags()                     # how far back a record is distinguishable
from honestboard import certify_chain
certify_chain(b, rule="lord", baseline="certified")   # sequential certification in date order
b.resolution()                      # minimum detectable difference, tasks needed for 1 and 2 points
b.winner_adjusted()                 # selection-corrected score for the leader (iterated bootstrap)
b.check_claim("some-system-name")   # the main facts about one system in one call

Command line:

python -m honestboard report swebench_verified.json
python -m honestboard check swebench_verified.json 20251205_sonar-foundation-agent_claude-opus-4-5
python -m honestboard pr-check experiments/evaluation/verified experiments/evaluation/verified/NEW_SUBMISSION --tasks verified_ids.json

pr-check reads a checkout of a leaderboard repository and prints a Markdown report on one submission (rank interval, margin over the current record, a paired test of the claim, what the board can resolve). examples/leaderboard-pr-check.yml shows how a leaderboard repository can post that report on every pull request.

Board file (JSON): {"tasks": [...ids...], "systems": [{"name": "...", "date": "20251205", "resolved": [...ids...]}], "clusters": {task: group}}. For repeated trials use "rates": {task: share_passed} instead of "resolved". A CSV with header name,date,<task ids> and 0/1 or rate cells also works. clusters is optional; cluster=True needs it and raises an error without it.

Methods

Limits you should know about

• It measures task-sampling luck only. Developers tuning on a public benchmark and run-to-run randomness of agents are outside the model, so the real overstatement of a winner is larger than the correction removes.
• Tasks are treated as independent draws. Tasks from the same repository are correlated; cluster=True is a rough sensitivity check (with few clusters it is not reliable and the paired tests still ignore clusters).
• The record test conditions on the record event (the selective p-value); it is exact for a fixed holder under the task-wise null and conservative when the previous record is a running maximum (under a global null it rejects well under 5% of record steps). A record that does not pass is not necessarily worse. It cannot be told apart from the previous record with these tasks. The task-level test treats tasks as independent; where tasks share a repository, cluster=True is the cautious count.
• The winner correction removes most of the bias in simulations but not all of it, much less under a shift to other repositories, and its interval is approximate (below nominal at small sample sizes).
• The conditional method is exact under normality but gives an unbounded answer when the winner ties or nearly ties the runner-up.
• The record chain depends on how submissions with the same date are ordered.

Install and test

pip install honestboard   # from PyPI (numpy and scipy only); or, from a checkout of this folder:
pip install -e .
python -m pytest tests

MIT licensed. If you use honestboard, please cite the paper (see CITATION.cff).