claimeval

Uncertainty quantification for cost–accuracy claims about AI agents.

Leaderboards, press releases, and papers declare:

• "Agent A is #1 on SWE-bench Verified"
• "Model X outperforms Model Y"
• "System A matches System B at a fifth of the cost"
• "Agent A is Pareto-optimal on the cost–accuracy frontier"

Each of these is a statement about noisy sample estimates from a finite benchmark of tasks and runs. claimeval attaches a probability to each such claim, using a bootstrap over the correct unit of generalization, and reports whether it is statistically supported.

---

🚀 Quick Start

Installation

claimeval requires only NumPy:

pip install claimeval

Or install from source in editable mode:

git clone https://github.com/claimeval/claimeval.git
cd claimeval
pip install -e .

---

🔍 Test a Claim in One Command

Evaluate claims against a raw rollout log:

claimeval claims examples/runs.jsonl examples/claims.json --map examples/map.json

Output:

id        kind            subject                      P(claim)  verdict
c1        top1            agent-E                         0.960  supported
c2        pairwise        agent-D                         0.996  supported
c3        cost_dominance  agent-D                         0.717  NOT supported
c4        frontier        agent-A                         0.993  supported
supported = P >= 0.9; resampling: task

Claim Definitions (claims.json)

Define claims in testable JSON schemas:

[
  {
    "id": "c1",
    "kind": "top1",
    "subject": "agent-A",
    "comparators": ["agent-B", "agent-C"]
  },
  {
    "id": "c2",
    "kind": "pairwise",
    "subject": "agent-A",
    "comparators": ["agent-B"]
  },
  {
    "id": "c3",
    "kind": "cost_dominance",
    "subject": "agent-B",
    "comparators": ["agent-D"]
  },
  {
    "id": "c4",
    "kind": "frontier",
    "subject": "agent-A",
    "comparators": ["agent-B", "agent-C", "agent-D"]
  }
]

Supported claim types:

• top1: The subject strictly leads all comparators.
• topk: The subject ranks within the top $k$ systems.
• pairwise: The subject strictly outperforms named rivals.
• parity: Performance difference is within a threshold ($|a_c - a_f| \le \delta$).
• cheaper_noninferior: Subject is cheaper than rival and accuracy is within parity margin.
• frontier: Subject belongs to the non-dominated Pareto frontier.
• cost_dominance: Subject is both strictly cheaper and more accurate than rival.

---

🐍 Python API

from claimeval import Rollout, RolloutSet, Claim, evaluate_claims

# Ingest rollout records: (config, task_id, seed, success_bool, cost_dollars, group)
rollouts = [
    Rollout("agent-A", "task-1", seed=0, success=1.0, cost=0.15, group="repo-1"),
    Rollout("agent-B", "task-1", seed=0, success=0.0, cost=0.08, group="repo-1"),
    # ...
]
dataset = RolloutSet(rollouts)

# Test a pairwise claim
claims = [Claim("pairwise", subject="agent-A", comparators=["agent-B"])]
results = evaluate_claims(dataset, claims, threshold=0.9, n_boot=2000)

for r in results:
    print(f"Claim {r['id']}: P = {r['probability']:.3f} -> {r['verdict']}")

Modular Architecture & Subpackages

claimeval is organized into clean, focused subpackages:

• claimeval.core: Data models (Rollout, RolloutSet), stream/file ingestion (load, inspect_path), and benchmark loaders (swebench, terminal_bench, tau2).
• claimeval.stats: Resampling estimators (hierarchical_bootstrap, variance_decomposition), Pareto geometry (dominance, pareto_mask, frontier_band, hausdorff), cluster methods (wild_intervals), BCa intervals, bias corrections (split_sample_debias), and cost diagnostics (diagnose_cost_axis).
• claimeval.analysis: Claim evaluation engine (Claim, evaluate_claims), benchmark baselines, pairwise comparison suites, and formatted tabular reporting (analyze, headline).
• claimeval.sim: Generative ground-truth simulations (make_population, simulate) and Monte Carlo estimator coverage studies (coverage_study).

All common functions remain accessible directly from the top-level import claimeval.

---

📊 Auditing Public Benchmarks Directly

claimeval provides built-in loaders for public agent benchmarks:

# SWE-bench Verified (from github.com/SWE-bench/experiments)
claimeval claims evaluation/verified claims.json --from swebench

# SWE-bench Verified cost leaderboard (mini-SWE-agent runs)
claimeval claims evaluation/verified claims.json --from swebench-cost

# Terminal-Bench 2.0 (Harbor evaluation trees)
claimeval claims submissions/terminal-bench/2.0 claims.json --from terminal-bench

# tau2-bench leaderboard trajectories
claimeval claims tau2_results/ claims.json --from tau2

---

🛠️ CLI Toolkit

Command	Purpose
claimeval claims	Evaluate a set of typed claims against a rollout dataset
claimeval analyze	Compute empirical Pareto frontier with membership & dominance probabilities
claimeval inspect	Automatically inspect schema and statistics of an unknown evaluation dump
claimeval seed-curve	Simulate error of a $k$-run study vs. full data
claimeval coverage	Validate estimator calibration against known ground truth
claimeval demo	Run an end-to-end demonstration on synthetic benchmark data

---

👤 Author & Contact

Harish Mashetty
Independent Researcher
E-mail: harish.mashetty@gmail.com
Project Repository: https://github.com/claimeval/claimeval

---

📜 License

MIT License. See LICENSE for details.