evalkit

Per-query eval results and paired-bootstrap significance testing for retrieval and LLM systems.

Answers the question most eval harnesses can't: did that change actually improve anything, or did the number just move?

from evalkit import run, paired_bootstrap

baseline = run("baseline", search_v1, queries, qrels, ["recall@10", "ndcg@10"])
candidate = run("candidate", search_v2, queries, qrels, ["recall@10", "ndcg@10"])

v = paired_bootstrap(candidate, baseline, "ndcg@10")
print(
    f"{v.observed:+.4f}  95% CI [{v.ci_low:+.4f}, {v.ci_high:+.4f}]  p={v.p_value:.4f}"
)
# +0.0207  95% CI [+0.0106, +0.0320]  p=0.0003
if not v.significant:
    raise SystemExit("difference is within noise — don't ship on this evidence")

Install

uv add paired-evalkit
# or
pip install paired-evalkit

import evalkit  # the import name is `evalkit`, not `paired_evalkit`

The distribution is paired-evalkit; the import is evalkit. evalkit on PyPI is an unrelated project (evolutionary algorithms), so the published name carries the differentiator this package actually has. If you happen to have both installed they share the evalkit module name and will conflict — uninstall one.

To pin a specific commit instead of a release:

uv add "evalkit @ git+https://github.com/skbugudi/evalkit@<sha>"

Pin a rev rather than a bare URL. Unpinned, a resolve tracks whatever main holds, so an upstream merge changes your CI with no commit in your repo — and a metric-definition change makes previously saved baselines incomparable with nothing saying so.

Python 3.12+. Runtime dependencies: numpy, httpx, pydantic, pyyaml, typer, anthropic (the last used only by the LLM judge).

Why

1. Aggregates hide the thing you need to know. A mean tells you a system scores 0.7377. It can't tell you which queries it won or lost, so it can't tell you whether A beats B or merely averages higher. Two systems with identical means can disagree on every single query.

This isn't hypothetical. On real SciFact data a hybrid retriever beat a dense one on the aggregate (+0.0091 nDCG) while losing on more queries than it won — 14 wins, 16 losses, 70 ties. The aggregate said ship it. It was noise (p = 0.65).

2. "The number went up" is not evidence. A paired bootstrap resamples your query set with replacement 10,000 times and reports the spread. If the 95% interval straddles zero, a different sample of queries could plausibly have reversed your verdict.

3. Underpowered evals produce meaningless nulls. Simulated at n=100:

true effect	detected
+0.02	12% of the time
+0.05	60%
+0.10	100%

A 3-point improvement is missed more often than found at n=100. Knowing that before you run tells you whether the eval is worth running at all.

API

run(name, retriever, queries, qrels, metrics) -> RunResult

@dataclass
class QueryResult:
    query_id: str
    retrieved: list[str]
    metrics: dict[str, float]  # {"recall@1": 1.0, "ndcg@10": 0.63}


@dataclass
class RunResult:
    name: str
    per_query: list[QueryResult]  # always retained

    def mean(self, metric: str) -> float: ...  # aggregates are DERIVED

• retriever — any callable (query) -> list[str], ids best first.
• queries — any objects exposing .id. Bring your own type.
• qrels — dict[str, set[str]], query id to relevant ids.
• metrics — spec strings parsed against runner.METRICS: "recall@10", "ndcg@10". Add your own with METRICS["precision"] = precision_at_k. Note: this is the runner metric table, shape (retrieved, relevant, k) -> float. It is separate from the @register_metric registry used by the target/gate path — see "Two metric tables" below.

A query missing from qrels raises rather than scoring 0.0. Silently scoring an unjudged query as zero makes a broken eval look like a worse system, which sends you to debug the model when the harness is what broke.

paired_bootstrap(a, b, metric, n_resamples=10_000, seed=42) -> BootstrapResult

Returns observed, ci_low, ci_high, p_value, and significant, plus provenance (metric, name_a, name_b, n_queries, n_resamples).

Paired is load-bearing. It resamples query ids and looks up both systems' scores for those same ids. Query difficulty dominates the variance, and pairing subtracts it out before the statistics see it — comparing independently drawn samples throws away most of your power. Alignment is by query id, never by row position; there's a test asserting that shuffling one system's rows changes nothing.

Implementation details that matter:

• CI from percentiles of the resampled distribution.
• p-value from a centred null (resampled - observed), which simulates a world where the two systems are equivalent.
• p = (extreme + 1) / (n_resamples + 1) — floors p at 1/10001 rather than reporting exactly 0. Never seeing something in 10,000 draws is evidence it's rare, not impossible.

Cost: the bootstrap resamples results you already collected. It never re-runs a query, so 10,000 resamples cost nothing beyond CPU. Run the eval once, persist the per-query rows, bootstrap offline as often as you like.

corpus — optional BEIR loaders

load_corpus(), load_queries(), load_qrels() read BEIR-format JSONL and TSV. Convenience only; run() never requires them.

Known limitation: corpus.DATA resolves relative to the installed module, so the loaders only work from a source checkout with a sibling data/ directory. If you're bringing your own data — which you are, if you installed this as a package — build queries and qrels yourself. See examples/custom_client. Making the data root configurable is tracked as the first API change.

Extension points

A client project adds behaviour by decorating its own functions. Nothing in this package needs editing.

from evalkit import register_metric, register_aggregate, register_loader


# per-case: (TargetResponse, GoldenCase) -> float
@register_metric("precision", higher_is_better=True)
def precision(response, case) -> float:
    hits = sum(1 for i in response.ranked_ids[:3] if i in case.relevant)
    return hits / 3


# the direction that makes this one a regression is the opposite one
@register_metric("unsupported_claims", higher_is_better=False)
def unsupported_claims(response, case) -> float:
    return float(count_unsupported(response.text))


# whole-run: (Sequence[CaseResult]) -> float
@register_aggregate("cost_per_query", higher_is_better=False)
def cost_per_query(rows) -> float:
    return sum(r.response.meta.get("cost_usd", 0.0) for r in rows) / len(rows)


@register_loader("parquet")  # (Path) -> GoldenSet
def load_parquet(path) -> GoldenSet: ...

Duplicate names raise. A name may be a per-case metric or an aggregate, never both, so check resolution is unambiguous.

higher_is_better is required

It has no default, on purpose. Which direction counts as a regression is a property of the metric, not of the check that reads it — recall falling is a regression, an unsupported-claim count falling is an improvement, and max_item_share and entropy point opposite ways on the very same run. Nothing outside the metric can know.

Defaulting it to True would mean the first lower-is-better metric anyone registers silently gates backwards: a regression gate that passes the exact change it exists to block, while still reading as coverage. polarity(name) returns it; unknown names raise rather than assuming a direction.

Two metric tables — read this before registering anything

There are currently two, because the retrieval path predates the target path:

	shape	used by
runner.METRICS	(retrieved, relevant, k) -> float	run() with spec strings like "recall@10"
@register_metric	(TargetResponse, GoldenCase) -> float	the Target/gate path

Register into the one matching the path you're using. A metric registered in the wrong table is simply never found. These converge in a later release.

Targets and goldens

from evalkit import CallableTarget, HttpTarget, load_goldens

goldens = load_goldens("goldens/cases.jsonl")  # jsonl | yaml | csv

local = CallableTarget(my_ranker, name="local")

with HttpTarget(
    url="http://localhost:3001/api/search",
    name="prod",
    build_payload=lambda case: {"q": case.query},
    extract=lambda body: [r["id"] for r in body["results"]],
    meta_keys=("cost_usd", "my_custom_field"),
) as remote:
    response = remote.invoke(goldens.get("q1"))

HttpTarget retries once on a 5xx or a transport failure, never on a 4xx, and measures latency across the whole attempt sequence. It reuses one connection pool, so use it as a context manager (or call .close()).

Everything fails loudly: an unreachable target, a malformed golden, a JSON body that isn't an object, or a golden file that loads zero cases all raise rather than producing a zero that looks like a quality regression.

Built-in metrics

Importing evalkit registers these. registered() returns the live list.

Name	Kind	Polarity	Notes
recall@1/3/5/10/20, ndcg@10	per-case	higher better	adapts the verified core functions to (TargetResponse, GoldenCase)
latency_p50/p95/p99, latency_mean	aggregate	lower better	nearest-rank, not floor(n·p)
max_item_share, max_item_share@1/3/10	aggregate	lower better	fraction of queries an item appears in
entropy	aggregate	higher better	normalised to [0, 1] so catalogue sizes compare

Add a k without editing the package:

from evalkit.metrics.retrieval import register_retrieval_metric
from evalkit.metrics.core import recall_at_k

# registers "recall@50"
register_retrieval_metric("recall", recall_at_k, 50, higher_is_better=True)

Percentiles are nearest-rank on purpose. The common hand-written form sorted[floor(n * p / 100)] returns the maximum whenever n · p / 100 is an integer — at p95 that is every run size divisible by 20. A 20-case eval then reports its worst case and labels it p95.

max_item_share counts queries, not slots. If 100 queries return 3 items each and one item appears in 40 of them, its share is 0.40, not 0.13. The question "am I showing everyone the same thing" is about queries; a slot-count denominator dilutes exactly that signal. Ids are deduped within one response, so a target that repeats an id cannot inflate its own share.

LLM-as-judge

from evalkit import register_metric
from evalkit.metrics import LLMJudge

judge = LLMJudge(
    rubric="Score 5 if the answer is correct and complete, 1 if it is wrong.",
    model="claude-haiku-4-5",  # required — there is no default
    scale=5,
    cache_path=".evalkit/judge-cache.json",
)
register_metric("helpfulness", higher_is_better=True)(judge)

The model is a required argument. It is part of the eval's definition, not a runtime detail: swapping judges makes scores incomparable to a stored baseline, which is a silent baseline invalidation. It is folded into the cache key, so a swap is a cache miss rather than a reuse of the previous model's grades. Grading is a forced, strict tool call — there is no free-text parsing step that can fail partway through a paid run. Latency is deliberately not hashed, so rerunning an unchanged eval costs nothing.

Checks

A check declares either a fixed threshold or a comparison against the stored baseline — never both, never neither.

checks:
  - id: recall_floor
    metric: recall@10
    op: ">="            # "<=" | ">=" | "=="
    threshold: 0.60

  - id: p95_ceiling
    metric: latency_p95
    op: "<="
    threshold: 800
    level: warn          # "block" (default) | "warn"

  - id: no_ndcg_regression
    metric: ndcg@10
    vs_baseline: no_significant_regression
    alpha: 0.05

from evalkit import load_checks

specs = load_checks("evals/checks.yaml")

Declaring both leaves two verdicts for one check with no rule for combining them. Declaring neither is worse — a line in a config file that gates nothing while reading as coverage. Both raise at load time, as do an unknown key (a typo'd treshold would otherwise gate the wrong thing), a duplicate id, a file that loads zero checks, and an alpha on a threshold check, which has no significance test to apply it to.

Metric names are not resolved at load time — a config may legitimately name a metric the client registers later.

scope — gate on named cases

A check with no scope covers every case in the run. scope restricts it to the ones you name:

checks:
  # the cases that already work hold an absolute floor
  - id: controls_hold
    metric: ndcg@3
    scope: [cp-surry-any, cp-newtown-pub, cp-bondi-any]
    op: ">="
    threshold: 1.0

  # everything else, including the known-broken cases, only may not get worse
  - id: no_regression
    metric: ndcg@3
    vs_baseline: no_significant_regression

That pairing is the point. On a set where some cases are known-broken and baselined at zero, a single absolute threshold across everything is red on day one — and a gate that is red on day one is switched off inside a week. Scoped floors let the working cases hold a hard line while the broken ones are gated only on further degradation, so the gate is green today and still fails the moment something real regresses. As each defect is fixed you promote a new baseline and the old failures become the new floor.

It is also what keeps per-case gating out of every client. Without it, each project reimplements the same loop over per_query in its own runner, which is the boundary this package exists to hold: clients write their cases, their metrics and their thresholds — not their gate logic.

scope is orthogonal to the either/or above: it works on threshold and vs_baseline checks alike.

Two ways a scoped check can lie, and both raise.

Naming a case that does not exist is refused at load, when you pass the ids you have:

specs = load_checks("evals/checks.yaml", known_case_ids=goldens.ids())
# CheckError: scope names 1 case(s) that do not exist: cp-newton-pub

A typo'd case id is a config error, and a config error should surface when the config is read — not after a full scored run has been paid for. known_case_ids is optional, so a client that has not loaded its goldens yet can still read config; supply it wherever you can.

Passing on cases it never evaluated is refused at run, by select:

spec.select(result.per_query)
# CheckError: check 'controls_hold': scoped to 3 case(s) but 1 produced no
# result: cp-bondi-any. A scoped check must evaluate every case it names —
# a floor checked on a subset is a floor that was not checked.

Reaching that error means the case was known and produced no row — it errored, timed out, or the runner dropped it. Evaluating the survivors would report the same green as a full pass, and the difference is exactly the case that fell over. An empty scope: [] is refused outright, as is a repeated case id, which would be weighted twice.

Report spec.coverage(rows) alongside the verdict — controls_hold: pass (4/4 cases). A scoped check that prints only "pass" leaves the reader unable to tell a gate that held from a gate that matched nothing.

Comparison

<= and >= are exact. A threshold is a line, and a tolerance around it just moves the line somewhere undocumented.

== uses a fixed tolerance — math.isclose(observed, threshold, rel_tol=1e-9, abs_tol=1e-12). Exact equality would mostly work: essentially every legitimate == in an eval config is a zero-tolerance count (error_count == 0) or a boundary rate (pass_rate == 1.0), and nobody sanely writes ndcg@10 == 0.7377. The problem is not wrongness but a confusing failure — a report printing 0.0000 beside a check that failed on 1e-17. The abs_tol is the load-bearing half: isclose with only a relative tolerance never matches against zero, which is the main case.

The tolerance takes no argument and is not a config key. A per-check tolerance is a strictness dial, and a gate whose strictness is tunable per check is a gate someone quietly loosens under deadline.

For a zero-tolerance count, prefer op: "<=" with threshold: 0. It is unambiguous and sidesteps what == means on a float entirely.

  - id: no_unsupported_claims
    metric: unsupported_claims
    op: "<="
    threshold: 0

Example

examples/custom_client evaluates an HTTP-style client against a hand-written golden set, with no network. Run it:

uv run python examples/custom_client/evaluate.py

metric        baseline  candidate    delta
------------------------------------------
recall@1        0.2000     0.7000  +0.5000
ndcg@10         0.7750     1.0000  +0.2250

Paired bootstrap (10,000 resamples):
  recall@1   +0.5000  95% CI [+0.1000, +0.9000]  p=0.0203  REAL
  recall@5   +0.0000  95% CI [+0.0000, +0.0000]  p=1.0000  not significant
  ndcg@10    +0.2250  95% CI [+0.0899, +0.3566]  p=0.0006  REAL

Development

uv sync
uv run pytest          # 101 tests, all offline
uv run mypy src tests  # strict
uv run ruff check .

The four core modules are copied verbatim from a verified source and are deliberately byte-identical to it — ruff line-length is pinned to 88 so formatting never rewrites them.

The significance tests are worth reading before trusting the statistics: they construct systems whose answer is known in advance (one better on every query; a system compared against itself) and assert the bootstrap agrees. Calibration was also checked by simulation — 300 trials under a true null produced a 7.7% false-positive rate against a nominal 5%, which is the mild anti-conservatism the percentile bootstrap is known for at n≈100. Treat a borderline p = 0.04 with proportionate caution.

License

MIT