agentstats

Statistics for agent evals. Point it at your agent's runs (Harbor jobs, ATIF trajectories or a JSONL file) and it tells you how good the agent is, how reliable it is, whether version B is really better than A, and whether a cheaper model is shown to be no worse. Every number comes with a confidence interval, and the report says which claim the numbers support.

pip install agentstats        # Python 3.11+
agentstats summary jobs/run1 jobs/run2 jobs/run3
agentstats compare --a jobs/baseline --b jobs/candidate
agentstats judge labels.jsonl
agentstats gate --policy agentstats.toml --tier nightly --candidate results/ --baseline baseline/
agentstats convert-traces traces.json --format otlp --rules rules.json --out rows.jsonl   # or --format langfuse
agentstats preregister --name my-switch --margin 0.03 --provenance independent --out decisions/my-switch
agentstats switch --prereg decisions/my-switch/prereg.json --current current.jsonl --candidate cheap.jsonl \
    --out decisions/my-switch

Why

Agents are stochastic, so a single eval run is a sample, not a measurement. On 100 SWE-bench Verified tasks, the same agent with the same model and setup (mini-swe-agent, DeepSeek-V4.1-Flash) scored 48%, then 40%. Eight tasks flipped between identical runs. If you compare two versions on one run each, that noise is your result.

Three other things go wrong in practice:

• Infrastructure failures get scored as agent failures. A quota error or a billing outage looks like a bad agent. agentstats keeps them out of pass rates, counts them, and warns if one arm had more.
• The wrong interval for how the eval set was built. Most eval sets are hand-picked. Treating each case as an independent sample of real traffic gives intervals that are too narrow, and a claim the data can't support.
• An unchecked LLM judge. A lenient judge passes behaviour humans would reject, and that's how regressions ship.

What it does differently

Other good tools exist (see Related tools). agentstats is built around agent runs:

1. Reads trajectories, not score tables. Harbor job directories and ATIF trajectories directly: pass/fail, infrastructure errors, tokens, cost, latency, steps and tool calls, plus deterministic rule checks on the tool calls (called, not_called, before, args).

2. Picks the statistics from how the eval set was built, and says what claim they support.

   --design	Your eval set	Interval / comparison	Claim
   curated (default)	hand-picked cases, most eval sets	fixed cases: noise from repeated trials of each case (falls back to Student's t / Welch's t over complete runs when runs vary more than that predicts)	"on this eval set"
   sampled	drawn at random from real traffic	per-case t / paired per-case differences	"on the population the cases came from"
   clustered	sampled, in groups that fail together	cluster-robust standard errors	"on the population the groups came from"

   For a hand-picked set the cases are fixed, so the only randomness is the agent's own: the fixed-cases test measures it from repeated trials of each case, and cases that always (or never) pass contribute nothing. It's the paired test from a production diagnosis gate, plus a small-sample correction that brought simulated false alarms from 6.8% to 5.8% at a nominal 5% (the rest is the discreteness of pass/fail; a simulated threshold would close it). It checks for run-wide shocks (whole runs varying more than per-case randomness predicts, as when a provider has a bad hour) and then falls back to treating each complete run as one observation. With one run it uses a Wilson interval or an exact McNemar test. Overlapping groups can't be modelled credibly, so clustered refuses them and explains why.

3. Reliability as its own result. pass@k (capability) and pass^k (consistency) with CIs, and lists of always-pass, never-pass and flaky cases.

4. Gates that can say "I can't tell". Two regression rules: no_significant_drop fails only on a proven drop (for pull requests); no_regression passes only when the change is shown to be no worse than a margin (non-inferiority, for releases), so an under-powered run is INCONCLUSIVE, with an estimate of the runs that would settle it, never PASS. A run-wide shock with too few runs to correct for it is also INCONCLUSIVE ("rerun"). Infrastructure failures (Harbor exceptions or an infra_error field) are kept out of pass rates and make a run UNHEALTHY (exit 3), as eval-sprint also does.

5. Judge checks: human agreement (Fleiss' κ), judge vs humans (accuracy, strictness, leniency, Cohen's κ, each with a CI, on a held-out split if you mark one) and judge consistency over repeated runs (flip rate, mean pairwise disagreement).

6. Model switches decided by a rule written down first. agentstats switch answers SWITCH / KEEP / NOT YET for moving to a cheaper model: non-inferiority at your margin plus a cost-ratio limit, the model as-is and with an adapted harness, per-slice results, and a warning when the eval set came from the current model. agentstats preregister fixes the rule before any result exists, and the decision only counts as pre-registered if the registration was committed before the results and its inputs are unchanged. See Deciding a model switch.

Example: did giving the fixing agent a diagnosis help?

mini-swe-agent on 100 SWE-bench Verified tasks, once plain and once with a diagnosis from a separate diagnosis agent added to the task as a hint. One Harbor job each.

$ agentstats compare --a jobs-plain100/... --b jobs-hint100/... --a-name plain --b-name with-diagnosis

NO DETECTABLE DIFFERENCE: with-diagnosis minus plain = +1.0 pts (95% CI -6.2 pts to +8.2 pts),
on this eval set, from a single run per arm (indicative: run-to-run noise is unmeasured).
Method: exact McNemar test + paired differences. 100 matched cases; plain 48.0%, with-diagnosis 49.0%.

- McNemar (exact): 7 cases pass only with with-diagnosis, 6 only with plain; p = 1.
- Smallest difference this setup can reliably detect (80% power): about +10.2 pts.
- Cost per attempt: plain $0.0858, with-diagnosis $0.0765; difference $-0.0093 (95% CI -0.0182 to -0.0005).

The +1 point is noise: 13 tasks moved, in both directions, and this setup can't see anything smaller than about 10 points. The cost difference is real (its CI excludes zero): the hint made the agent about 11% cheaper.

The same plain setup, run twice:

$ agentstats summary jobs-plain100/... jobs-all500/...

Pass rate: 44.0% (95% CI 41.2% to 46.8%), on this eval set.
Method: fixed cases, noise from repeated trials per case. 100 cases, 2 run(s).
- Not used: Student's t over complete runs would give 44.0% (95% CI 0.0% to 94.8%) (for hand-picked sets
  with 3+ runs that show run-wide shocks).
- Note: Whole runs vary 8.0x more than per-case randomness predicts (p = 0.005): a run-wide shock. With fewer
  than 3 runs there's nothing safer to fall back to; treat this with care.

| k | pass@k (capability)           | pass^k (consistency)          |
| 1 | 44.0% (95% CI 34.5% to 53.5%) | 44.0% (95% CI 34.5% to 53.5%) |
| 2 | 48.0% (95% CI 38.0% to 58.0%) | 40.0% (95% CI 30.0% to 50.0%) |
Always pass: 40, never pass: 52, flaky: 8

48% and 40% is more than per-case randomness explains: something changed across the whole run between the two (they ran at different times, in jobs of different sizes). A narrow interval from these two runs would be false confidence, and the report says so. A third run lets it fall back to run-level statistics.

In production: an incident-triage agent's CI gate

agentstats runs the regression gate for a production incident-triage agent (classifies alerts as real / noise / duplicate with a P0-P3 severity): 114 held-out cases, 2 trials per pull request, compared case by case with a 4-trial baseline using the fixed-cases test, no_significant_drop at a 90% interval.

Sized by simulation from the baseline's own per-case rates (each case's true rate drawn from its baseline trials, then candidate and baseline simulated from it; agentstats size does the same for your baseline), against the gate it replaced (fail if more than 6% of 80 filtered cases miss, one run):

The old gate also had to drop 32 noisy cases to get under its threshold; the paired test keeps them, since each case is compared with its own baseline rate.

A skipped tool the labels rewarded. The triage agent (then on Claude Haiku 4.5) skipped a required lookup tool and wrote an invented count instead. The eval labels had been built from its own answers, so the accuracy gate couldn't see it. With behaviour rules, Haiku 4.5 as the candidate against Haiku 5.5: accuracy +0.9 pts (CI -2.1 to +3.9), PASS; min_rule_rate FAIL (lookup called in 25.4% of attempts, CI 21.0% to 30.5%); no_rule_regression FAIL (-74.6 pts vs 100%). On 61 real-incident traces exported from Langfuse and read with convert-traces (rules-only, no labels): the lookup was called in 52.5% (CI 40.2% to 64.5%). The rules now run in that agent's CI gate.

What it caught in its first week:

• An environment offset. Six CI trials of unchanged code averaged 89.9% against a baseline measured on a laptop (93.2%): -3.3 points, 95% CI -6.1 to -0.5, spread over many cases, no run-wide shock. One pull request had already failed for it. The baseline is now measured in CI; the first gate run against it: -0.7 points (CI -3.4 to +2.1), PASS.
• Outages, as UNHEALTHY instead of FAIL. Once when the API credit ran out mid-run (69 of 228 attempts failed: exit 3, the cause listed, the remaining attempts compared separately), once when a new major SDK release rejected a request parameter on every call.
• A doubtful "same setup". Two runs of the same coding agent on the same 100 tasks scored 48% and 40%: the shock check flagged it (whole runs varying 8x more than per-case randomness predicts, p = 0.005), so the narrow interval wasn't trusted (see the example above).

Deciding a model switch: agentstats preregister and agentstats switch

A cheaper model comes out. "No significant difference" on one eval run isn't "no worse": a small run usually just has a wide interval. agentstats switch asks the question the other way round, as a non-inferiority test: is the cheaper model shown to lose no more than your margin, at no higher cost? It answers SWITCH, KEEP or NOT YET and writes a decision record (decision.json + decision.md) with the rule, the numbers and hashes of the inputs.

agentstats preregister --name my-switch --margin 0.03 --provenance independent \
    --input labels=evals/labels.jsonl --slice-by template --out decisions/my-switch
git add decisions/my-switch && git commit -m "pre-register my-switch"     # before running anything
# ... run both models on the same cases, a few trials each, same session ...
agentstats switch --prereg decisions/my-switch/prereg.json \
    --current current.jsonl --candidate cheap.jsonl [--adapted cheap-adapted.jsonl] --out decisions/my-switch

• Pre-registration is checked, not trusted. With --prereg the rule comes from the file (contradicting flags are an error), and the decision only counts as pre-registered if the registration is committed and unchanged, was committed before every result file was last modified, and every declared --input still has its registered hash.
• Two steps: the cheaper model as-is, then with an adapted harness (--adapted, e.g. rows from harness-evolve export); the better decision wins.
• Slices (--slice-by) are reported per slice; --strict-slices makes the decision no better than the worst one.
• Provenance: --provenance current-model says the eval set was made or filtered with the current model, and the record warns that the comparison favours it.
• Cost per accepted result is reported for each arm. Exit codes: 0 SWITCH, 1 KEEP, 2 NOT YET, 3 UNHEALTHY.

In production: my triage agent moved from Claude Haiku 4.5 to Haiku 5.5 on a pre-registered decision: +2.2 pts (95% CI -0.7 to +5.1) at 0.09x the cost per accepted result. An earlier comparison had said 10 points worse; the gap came from labels that repeated counts the old model made up. Rule, rows and record: examples/switch-triage-haiku-5-5.

These two commands started as a separate package, stepdown. They're a thin layer on agentstats' own tests, so they live here for now. If the model-switch side grows into its own system (suggesting candidate models, adapting the harness with harness-evolve, deciding as runs accumulate, shadow runs on live traffic), it may become a separate package again.

Evals in CI, in tiers

agentstats init-ci writes a policy (agentstats.toml) and three GitHub workflows. You replace one step in each, running your agent, and agentstats decides:

Behaviour rules as a regression gate (no labels needed). min_rule_rate requires every trajectory rule (the checks in each row, e.g. {"called_lookup_tool": true}) to hold at least that often, judged on its interval: FAIL only when the whole interval is below, INCONCLUSIVE when it can't tell. no_rule_regression compares each rule with the baseline, case by case (fixed-cases test, Bonferroni across rules), and fails when a rule is proven to hold less often. These catch what an accuracy gate can't when the labels are wrong: a triage agent that skipped a required lookup tool half the time and invented the number passed every accuracy check, because the labels had been built from its own answers. The gate prints a per-rule table either way.

Two more requirements, both from real misses: max_no_answer caps the share of attempts with no usable answer, and max_slice_drop (with slice_by = "<tag prefix>" in the tier) fails when any slice, e.g. one error template, is proven worse than the baseline by more than the limit. Slice intervals are Bonferroni-adjusted across slices, slices whose estimate is past the limit but not proven are named as leads, and with slice_by set the per-slice table is printed even without the requirement.

agentstats gate --policy agentstats.toml --tier nightly --candidate results/ --baseline baseline/ exits 0 pass, 1 fail, 2 inconclusive, 3 unhealthy (too many infrastructure failures to judge the agent), and writes its table to the GitHub job summary.

Size a gate before trusting it: agentstats size --baseline baseline.jsonl --trials 2 --rule drop simulates the rule on your baseline's own per-case rates and prints how often it would pass or fail for a range of true drops, so you can pick trials and margins (e.g. no change passes >= 90%, an 8-point drop fails >= 90%).

Comparisons with the baseline are non-inferiority tests. "The interval includes zero" doesn't show that nothing regressed, so no_regression = 0.02 passes only when the lower confidence bound of (candidate − baseline) is at least −2 points. An under-powered run is INCONCLUSIVE, with an estimate of how many runs would settle it, never PASS. Small smoke tiers say what they can't see: "with 20 cases and 1 run, this tier can only see drops larger than about 40 points: it catches breakage, not regressions."

On the real runs above (with-diagnosis as the candidate, plain as the baseline):

## agentstats gate: nightly ⚠️ INCONCLUSIVE

| requirement | measured | threshold | result | detail |
| runs | 1 | >= 3 | ⚠️ INCONCLUSIVE | the nightly tier expects 3 complete runs |
| max_infra | 0.0% | <= 5.0% | ✅ PASS | |
| no_regression | +1.0 pts (CI -6.2 pts to +8.2 pts, mcnemar) | lower bound >= -2.0 pts | ⚠️ INCONCLUSIVE | about 6 runs per arm would settle it if the estimate holds |
| max_cost_ratio | 0.89x (CI 0.80x to 0.99x) | <= 1.10x | ✅ PASS | |

Input

One row per attempt (agentstats convert writes this from Harbor jobs):

{"arm": "baseline", "run": 1, "case": "django-11099", "passed": true, "infra_error": null,
 "cost_usd": 0.12, "input_tokens": 41000, "output_tokens": 2100, "steps": 14, "tool_calls": 9,
 "latency_s": 33.0, "tags": ["feature:search"], "checks": {"no_refund_before_lookup": true}}

A run is one full pass over the eval set. Several paths for one arm are several runs. Runs are compared on the cases attempted in every run.

no_answer (optional, a reason such as "step_limit" or "parse_error") marks an attempt where the agent produced no usable answer at all. It counts as a failure, like a wrong answer, but is reported apart: a jump in no-answer attempts usually means the harness or the agent loop broke, not that the agent's judgement got worse. Rows are validated on load (the error names the file, line and field), and an optional schema_version (currently 1) lets a newer format be refused instead of half-read. harness-evolve export and agent-platform's triage gate both write this format; tests/test_hardening.py loads samples from each.

From traces (agentstats convert-traces): an OpenTelemetry OTLP JSON export (tool calls are spans with gen_ai.operation.name = "execute_tool", name in gen_ai.tool.name, arguments in gen_ai.tool.call.arguments; tokens from gen_ai.usage.*) or a Langfuse export (traces with their observations; TOOL observations are tool calls). One trace is one attempt; --rules fills its checks. Pass/fail comes from --verdicts (JSONL with trace_id, passed, optional case, run, infra_error); without it the rows are rules-only: no pass rate, but the rule gates work, which is what production traces without labels need.

Judge labels (agentstats judge), one row per verdict:

{"item": "conv-07", "rater": "human:alice", "verdict": "pass", "split": "heldout"}
{"item": "conv-07", "rater": "judge", "verdict": "fail", "run": 2}

Trajectory rules (agentstats check JOB --rules rules.json --out results.jsonl): see examples/rules.json. A synthetic judge-labels file is in examples/judge_labels.jsonl.

Methods and sources

• Error bars for evals, clustered standard errors: Miller 2024, arXiv 2411.00640.
• Choosing the method by how the eval set was built, evals in CI tiers, judge alignment and consistency metrics: Yesha Shah's "Agentic Evals" series (see Credits).
• pass@k: Chen et al. 2021, arXiv 2107.03374. pass^k: τ-bench, Yao et al., arXiv 2406.12045. Randomness in agentic evals: Bjarnason et al. 2026, arXiv 2602.07150.
• Judge self-inconsistency: Haldar & Hockenmaier 2025 (EMNLP Findings); Yagubyan 2026, arXiv 2606.13685.
• Pre-registration (fix the analysis before seeing the data): Nosek et al. 2018, "The preregistration revolution", PNAS. The switch rule is a standard non-inferiority test (lower interval bound above -margin) on the same fixed-cases interval the gates use.
• Cost per accepted result rather than cost per call: from Grep AI's "The Managed Agent Wars".

Every statistic is tested against scipy or statsmodels (tests/test_stats.py, tests/test_judge.py). The only runtime dependency is scipy.

Credits

The framing behind several features comes from Yesha Shah's Agentic Evals series: Part 1, "A Software Engineer's Guide" (2026-09-21); Part 2, "Statistical Tools for Measuring Agent Performance" (2026-09-30); Part 3, "Introduction to Autoraters" (2026-10-19):

• choosing the statistics by how the eval set was built (benchmark vs generalised performance; curated, sampled and correlated eval sets);
• pass@k and pass^k for agents, and reporting intervals rather than single runs;
• evals in CI in tiers (smoke, nightly, release), a stratified smoke sample, and rule-based checks on trajectories;
• the judge checks: strictness, leniency, Cohen's and Fleiss' κ, a held-out split, verdict flip rate and mean pairwise disagreement.

The statistical methods themselves come from the papers listed under Methods and sources, which the series also cites.

Original to agentstats:

• the fixed-cases test with its small-sample correction (from a production diagnosis regression gate), and the run-wide shock check;
• the gate rules: no_significant_drop and non-inferiority, INCONCLUSIVE with a runs estimate, a rerun on a shock;
• gate sizing by simulation on your own baseline (agentstats size);
• behaviour rules as a regression gate: no_rule_regression compares each trajectory rule with the baseline, case by case with intervals, and min_rule_rate sets an interval-based floor. They need no labels, which is what caught a production bug an accuracy gate passed (see the production section);
• the no-answer outcome (a broken loop or unparseable output reported apart from wrong answers) and per-slice gates with Bonferroni-adjusted intervals and named leads;
• rules-only rows from traces: OpenTelemetry (GenAI conventions) and Langfuse exports, so behaviour can be gated on production traces that have no labels;
• model-switch decisions (agentstats switch): non-inferiority plus cost per accepted result, as-is and adapted steps, an eval-set provenance warning, and checked pre-registration (committed before the results, inputs hashed), with a decision record;
• calibration tests that fail if the statistics drift (e.g. the drop gate's false-alarm rate);
• reading Harbor jobs and ATIF trajectories, and the production results above.

UNHEALTHY on infrastructure failures and keeping them out of pass rates are also in eval-sprint.

Related tools

• evalstats (arXiv 2609.35815): judge-corrected tests (PPI) and methods tuned for small samples. Use it when your verdicts come from an LLM judge with a small human-labelled set.
• evalci (arXiv 2607.04429): CIs, paired tests, power analysis and multiple-comparison correction for per-item tables (lm-eval, HELM).
• eval-sprint: an eval kit with deterministic and LLM judges, a labelling UI, and a CI regression gate (exact McNemar on paired flips, errors kept out of pass rates, an unhealthy-run exit code). Its judge calibration (blind expert labels, Horvitz-Thompson weights, Rogan-Gladen correction) is more thorough than agentstats' judge check.

Limitations

• Pass rates need pass/fail verdicts; numeric scores are used for efficiency metrics and judge consistency.
• Power analysis is by simulation (agentstats size) for the two gate rules; no closed-form sample-size planner.
• The judge check uses the judge's most common verdict per item; it does not correct downstream statistics for judge error (evalstats does).
• agentstats switch decides on fixed batches: run both models, then decide. Deciding as runs accumulate (anytime-valid intervals) isn't built. The margin is your judgement; agentstats makes you state it and checks you stated it first. Cost is whatever your rows record (my runs: token counts at list prices, no cache discounts).

MIT licence.