evalpower 0.1.0

How many runs before your eval means anything?

Your eval ran once and passed. What is the probability that means nothing?

Most eval harnesses answer a question nobody asked: did it pass this time? If the thing you are measuring is stochastic — and agent behaviour, tool use, and anything with a temperature above zero is stochastic — then a single green run is a sample of size one, and a suite of green runs is a suite of samples of size one.

This library takes the pass/fail counts you already have and tells you what they actually support:

• The k=1 audit miss rate. Of the cells you know fail, because you watched them fail, how often does a single run look clean anyway? Above 0.5 and one-shot auditing is worse than a coin flip.
• Whether a cell is demonstrably stochastic, under an exact Clopper-Pearson criterion that requires the interval endpoints to clear the always-pass and always-fail traps — not just the point estimate to look intermediate.
• How many runs you need to reach a given confidence, and its inverse: what a clean sweep of n runs actually rules out.
• pass^k and safe^k curves, computed unbiasedly rather than by raising an observed rate to the k-th power.

No containers, no adapter, no environment reset, and nothing to integrate. It runs against eval output you already have.

This is not significance testing. If your question is whether two models differ, you want a significance tool, and good ones exist for LLM evals. This one asks the prior question: whether a single model–task result is repeatable at all. Significance compares between arms; reliability lives within a cell.

Install

pip install evalpower

Zero runtime dependencies. Pure stdlib, and intended to stay that way: this drops into any pipeline without touching its dependency resolution, and there is no version of scipy that can break someone's build because they wanted a confidence interval.

Typed (py.typed, checked under mypy --strict), Python 3.11+, tested on 3.11 through 3.14.

Quickstart

from evalpower import CellStats, audit_miss_rate, clopper_pearson, runs_needed

# A "cell" is whatever you repeat runs over — usually (model, task).
cells = {
    ("gpt-x", "refund-flow"):   CellStats(n=16, x=4),   # 4 of 16 runs failed
    ("gpt-x", "ledger-close"):  CellStats(n=16, x=0),   # clean sweep
    ("claude-y", "refund-flow"): CellStats(n=32, x=12),
}

audit_miss_rate(cells)                  # 0.6875 pair-weighted
audit_miss_rate(cells, "event")         # 0.65625 weighted by failure events
clopper_pearson(4, 16)                  # (0.0727, 0.5238) — exact, not Wald
runs_needed(0.25, confidence=0.95)      # 11

That clean sweep is the interesting one:

from evalpower import zero_failure_upper_bound

zero_failure_upper_bound(16)   # 0.1707

Sixteen spotless runs do not show the failure rate is zero. They show it is probably below 17%.

From a file

$ evalpower results.csv

3 cells, 64 runs, 2 failure-producing, 16 failure events

  cell                    x/n    rate   95% CI (exact)  miss@1  stochastic  runs@95%
  ----------------------  -----  -----  --------------  ------  ----------  --------
  claude-y / refund-flow  12/32  0.375  (0.211, 0.563)  0.625   yes         7
  gpt-x / refund-flow     4/16   0.250  (0.073, 0.524)  0.750   yes         11
  gpt-x / ledger-close    0/16   0.000  (0.000, 0.206)  1.000   -           <=17.1%

k=1 audit miss rate over the 2 failure-producing cells
  pair-weighted   0.6875
  event-weighted  0.6562
  (the chance one run of a known-bad cell looks clean; above 0.5 is worse than a coin flip)

Demonstrably stochastic (exact 95% CI strictly inside (0.05, 0.95)): 2 of 3 cells
  carrying 100.0% of all failure events
  100.0% of failure events sit on cells with a rate in (0.1, 0.9)

  k    pass^k   safe^k
  ---  -------  -------
  1    0.79167  0.79167
  2    0.64435  0.64435
  3    0.54090  0.54090
  5    0.41944  0.41944
  10   0.33704  0.33704

The input is a table of per-cell counts, JSONL or CSV:

model,task,n,x
gpt-x,refund-flow,16,4
gpt-x,ledger-close,16,0
claude-y,refund-flow,32,12

Column names are matched leniently (n/runs, x/failures, …); see docs/concepts.md.

Or straight from per-run records

Harnesses emit one row per run, not per cell, so --per-run aggregates for you. No pre-processing script, which is otherwise the first thing standing between you and a report:

model,task,failed
gpt-x,refund-flow,true
gpt-x,refund-flow,false
gpt-x,refund-flow,false
gpt-x,refund-flow,false

$ evalpower --per-run runs.csv --k 1 2

1 cell, 4 runs, 1 failure-producing, 1 failure event

  cell                 x/n  rate   95% CI (exact)  miss@1  stochastic  runs@95%
  -------------------  ---  -----  --------------  ------  ----------  --------
  gpt-x / refund-flow  1/4  0.250  (0.006, 0.806)  0.750   -           11

k=1 audit miss rate over the 1 failure-producing cell
  pair-weighted   0.7500
  event-weighted  0.7500

Note what four runs buy: the interval is (0.006, 0.806), the cell does not clear the demonstrably-stochastic bar, and it would take 11 runs to be 95% confident of seeing that failure at all.

The outcome column can be named for either polarity — failed, passed, is_success — or be a neutral status/outcome/result holding pass/fail. Values may be true/false, yes/no, y/n, 1/0. Anything that could mean either thing (a neutral column holding true) is an error naming the flag that resolves it, never a guess. Override discovery with --cell-fields, --failure-field, --success-field.

In a pipeline

--json emits the same report as data, so you can gate on it:

$ evalpower results.csv --json | jq '.audit_miss_rate.pair'
0.6875

There is deliberately no --fail-under and no exit-code policy. What counts as an acceptable miss rate is a judgement about your system, not something this tool should hold an opinion about — so it always exits 0 and hands you the numbers.

Why exact intervals

Eval cells live where normal approximations fail: small n, proportions near 0 and 1. A Wald interval on 0 failures in 16 runs has zero width — it reports that you have measured the failure rate to be exactly zero, with certainty. Clopper-Pearson gives (0, 0.206). Every interval here is exact, computed by bisection on the binomial tail, with no continuity correction and no normal approximation anywhere.

The same discipline applies to the "how many runs" question. The rule of three (p <= 3/n given zero failures) overstates the limit by about 8 percentage points at n=7. This library uses the exact one-sided form, 1 - alpha^(1/n).

Provenance

These estimators are the measurement core of a reliability study of LLM agents, ported here with the domain-specific vocabulary generalized — that work's "damage" is this library's outcome-neutral "failure". The math is unchanged.

tests/test_paper_validation.py reproduces that paper's published figures exactly from its released run data — the audit miss rates, the trap-prevalence bounds, and the beta-binomial ICCs — so "this is the same computation" is a checkable claim rather than an assertion, and any future drift between the two fails a test. The test skips cleanly when the source data is not present, which is the normal case for everyone but its author.

Citation. The figures this library is validated against come from arXiv:2608.15286:

@misc{khurdi2026agentrelbench,
  title  = {No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage},
  author = {Shiven Khurdi},
  year   = {2026},
  eprint = {2608.15286},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url    = {https://arxiv.org/abs/2608.15286},
}

Scope

This library does statistics on counts. It does not run your evals, sandbox them, reset environments, or decide what "failure" means — that last one is yours, and the library is deliberately agnostic to it. If your labeller emits a conservative alternate count for ambiguous runs (crashes, timeouts), pass it as x_upper and it stays a separately reported bound, never fused into the headline.

Licence

Apache-2.0.