abeval

A/B-test statistics for LLM evals.

Eval scores are sample estimates, but they're routinely reported as exact numbers: "our prompt scores 76.5%, the old one 63.5%, ship it." On 200 items much smaller gaps than that are pure noise. abeval treats an eval like an A/B test — confidence intervals for a single run, paired significance tests for comparing two runs on the same items, and sample-size planning before you spend on inference. Standard library only, no dependencies.

Sample-size calculator: https://mohammadi.cv/abeval/

Install

pip install abeval

Or from source: git clone https://github.com/mohammadi-hadi/abeval && cd abeval && make install.

Is my new prompt actually better?

Point abeval compare at two JSONL result files that share item ids:

$ abeval compare examples/run_a.jsonl examples/run_b.jsonl
A: 63.5%   B: 76.5%   (n=200 paired items)
B - A: +13.0%  [4.5%, 21.5%]  (95% CI)
p (sign-flip permutation): 0.0041   p (paired t): 0.0034
discordant items: B-only wins 53, A-only wins 27   p (exact McNemar): 0.0049
verdict: significant at the chosen level

The comparison is paired: both runs are scored on the same items, so item difficulty cancels out and you detect much smaller differences than two independent runs would allow. Items present in only one run are dropped (and reported).

Every command takes --json for machine-readable output, and --seed makes the resampling reproducible.

Error bars for one run

$ abeval ci examples/run_a.jsonl
score: 63.5%  [56.6%, 69.9%]  (n=200, 95% CI, wilson)

Binary metrics get a Wilson score interval, continuous metrics a t interval (or --bootstrap). If your items aren't independent — 5 questions generated from each source document, say — pass --cluster-key document and the interval widens to account for within-cluster correlation.

How many items do I need?

Run this before the eval, not after:

$ abeval power --baseline 0.75 --delta 0.03
to detect 3.0% difference from a 75.0% baseline (corr=0.5):
  n = 1568 paired items  (80% power, 95% confidence)
  with  100 items you can detect >= 11.9%
  with  200 items you can detect >= 8.4%
  with  500 items you can detect >= 5.3%
  with 1000 items you can detect >= 3.8%

The lesson generalizes: a 200-item eval cannot see a 3-point improvement. --corr is the item-level correlation between the two runs' outcomes (0.5 is a reasonable default for two variants of the same system; higher correlation means fewer items needed). There's an interactive version at https://mohammadi.cv/abeval/.

How noisy is my judge?

If an LLM judge scores the same item differently on repeat calls, that noise eats your statistical power. Feed repeated judgments to reliability:

$ abeval reliability examples/judge_repeats.jsonl
40 items, 120 judgments (3.0 per item)
ICC (signal share of variance): 0.764
between-item sd: 1.708   judge noise sd: 0.9487
exact agreement across repeats: 15.0%
repeats per item to push judge noise under 10%: 3

ICC is the share of score variance that is real item signal rather than judge noise, from a one-way random-effects decomposition. The last line answers the practical question: how many repeat judgments to average per item.

Data format

JSONL, one item per line, any extra fields ignored:

{"id": "item-001", "score": 1}
{"id": "item-002", "score": 0, "category": "reasoning"}

--metric and --id-key rename the fields; booleans are accepted as 0/1. The files in examples/ are generated by examples/make_fixtures.py (seeded, so they're reproducible).

Python API

Everything the CLI does is a plain function:

from abeval import paired_compare, proportion_ci, sample_size, sd_diff_from_rates

proportion_ci(127, 200)                  # Wilson interval, level=0.95
result = paired_compare(scores_a, scores_b, seed=0)
result.diff, result.ci_lo, result.ci_hi, result.p_permutation

sd = sd_diff_from_rates(0.75, 0.78, corr=0.5)
sample_size(0.03, sd)                    # -> 1568 paired items

What's inside

Question	Method
CI for a pass rate	Wilson score interval
CI for a mean score	Student t, percentile bootstrap, or cluster-robust t
Is B better than A?	Sign-flip permutation test on paired differences (primary), paired t, exact McNemar on discordant pairs (binary)
How many items?	Normal-approximation power for the paired design, with a Monte-Carlo check (power_simulated)
Judge noise	One-way random-effects ICC with unbalanced-design correction

The permutation p-value is the primary test: it's exact under item exchangeability and makes no normality assumption. The paired t and McNemar values are printed alongside because reviewers ask for them.

The approach follows Miller, "Adding Error Bars to Evals" (2024) — report standard errors, use paired designs, plan sample sizes — packaged as a tool you can point at result files.

Honest limitations

• The power formulas use the normal approximation; for very small samples or rates near 0/1 trust power_simulated over the closed form.
• Cluster-robust intervals are approximate with fewer than ~20 clusters.
• Multiple-comparison correction is on you: if you compare ten prompts and ship the best p-value, it's inflated (a sweep command with Bonferroni correction is on the roadmap).
• This is not a stats library. If you need regressions or GLMs, use statsmodels; abeval covers the eval loop with zero dependencies.

Sponsoring

abeval is MIT-licensed and dependency-free, and it stays that way. Sponsoring funds the roadmap below and the maintenance time to keep the statistics trustworthy. Sponsors are credited in release notes and vote on what lands next: GitHub Sponsors.

Roadmap

• sweep: compare N runs against a baseline with multiple-comparison correction.
• Confidence intervals for pairwise win rates (Bradley-Terry) for head-to-head judge outputs.
• Variance-reduction guidance: how much power you buy by averaging k judge repeats, given your measured ICC.

Related projects

• judgewatch — monthly bias audits of LLM judges; abeval tells you whether this month's change is real.
• judgekit — audit your own judge pipelines.
• trajectory-judge — evaluating agent trajectories with LLM judges.

License

MIT — see LICENSE.