rag-eval-gate

Retrieval evaluation that tells you what it can't resolve.

Recall@5 = 1.00 on 10 queries is really [0.69, 1.00]. That +0.30 lift over BM25 has a p-value floor of 0.25 — no outcome of that comparison could ever have been significant. Both of those are computable from the published table alone.

→ Check a number right now, nothing to install

Type in the figure and the query count. It returns the interval that figure supports and, for a claimed improvement, whether any outcome of that comparison could have reached significance. The arithmetic runs in the page; nothing is sent anywhere.

Or from a terminal:

pip install retrieval-eval-gate      # the PyPI name; `rag-eval-gate` there is someone else's

rag-eval-gate audit -n 10 -v 1.00 -b 0.70     # audit a table you are reading
rag-eval-gate power gold.jsonl                # what your own gold set can resolve

Zero dependencies. Works with any retriever in any language. Runs as a GitHub Action. The command is rag-eval-gate; the distribution is retrieval-eval-gate because the shorter name on PyPI belongs to an unrelated project. Both spellings work as the command.

Three things your eval is probably not telling you

A perfect score is not a perfect score. Ten queries, one relevant document each, Recall@5 = 1.00 — that is ten successes out of ten, and the exact binomial interval bottoms out at (α/2)^(1/n) = 0.69. The headline is 31 points wide.

Some comparisons cannot be won. Your hybrid retriever beats BM25 by +0.30 Recall@5. On ten queries that is three queries flipping, seven tying — and a paired randomization test on three differing pairs has a smallest attainable two-sided p of 2/2³ = 0.25. Not "not significant": unable to be, whatever the result.

Your interval breaks when you look at it. A 95% interval promises that one look at a finished sample contains the truth 95% of the time. Watching a CI log is not that. Over 800 simulated evaluation streams, inspection starting only at n = 30 where the normal approximation is defensible:

Runs where the true mean escapes a nominal 95% interval at some point while it is watched. The fixed-N column grows with stream length — the longer you look, the more chances to be wrong. That is the law of the iterated logarithm, and it is why the right-hand column exists.

Audit a table in five seconds

No gold set, no retriever, no data — the reported means and the query count are enough:

$ rag-eval-gate audit -n 10 -m recall@5 -v 1.00 -b 0.70

recall@5 = 1.000 on 10 queries
  that is 10/10 successes

  95% interval   [0.692, 1.000]   width 0.308   (exact binomial)
  The reported figure is 31 points wide. Quoting it to two decimal places implies a
  precision 10 queries cannot deliver.

Claimed lift +0.300 over 0.700
  note: the two means determine it exactly: 3 of 10 queries differ

  smallest attainable p   0.2500

  UNRESOLVABLE. Under every possible overlap of the two systems' successes,
  a paired test on this many differing queries has a floor above 0.05, so no
  outcome of that comparison could have reached significance. This is a fact
  about the sample size, not about the systems.

It exits non-zero when the comparison could not have been significant, so it works as a check and not only as something to read. The interval is exact for a 0/1 metric; for a graded one the per-query spread is unknown, so it reports the widest interval the mean allows — a [0,1] variable with mean m has variance at most m(1−m).

The same thing runs in a browser at jinwovo.github.io/rag-eval-gate, with a shareable link for the result.

Quickstart

pip install retrieval-eval-gate

# 1. What can this gold set even measure?  (no retriever needed)
rag-eval-gate power gold.jsonl

# 2. Score it, with intervals and significance
rag-eval-gate eval gold.jsonl --modes bm25,vector,hybrid

# 3. Gate a pull request
rag-eval-gate eval gold.jsonl --gate --min-mrr10 0.85

A gold set is JSONL:

{"query": "how do I rotate a signing key", "relevant_doc_ids": ["kms-rotation"]}

…or the TREC qrels you already have (--gold qrels/test.tsv --queries-file queries.jsonl).

Works with whatever you built

Three ways in, so the language and shape of your retriever are irrelevant.

HTTP — a URL template and a small field map. No assumption about the route or schema:

rag-eval-gate eval gold.jsonl \
  --url-template 'https://search.internal/v2/query?text={query}&strategy={mode}' \
  --results-path 'data.hits' --id-field 'document_id' --score-field 'relevance'

A command — anything that prints JSON to stdout. Go, Rust, a notebook, a shell script:

rag-eval-gate eval gold.jsonl \
  --backend command --command './retrieve --q {query} --mode {mode} --json'

A TREC run file — rank the corpus once, offline, on whatever hardware; evaluate as often as you like with nothing running:

rag-eval-gate eval qrels/test.tsv --backend run-file --run-file runs/bm25.trec

Four gate policies

rag-eval-gate eval gold.jsonl --gate --gate-policy sequential

regression is the sensitive one. An absolute threshold only notices a regression once the mean crosses a line someone guessed; a paired test compares the same queries before and after, so a real drop on two of them is caught while the mean still clears the line.

sequential is the cheap one. Because a confidence sequence is valid at every sample size, stopping as soon as the verdict is settled is a decision rather than a peek:

undecided fails. The budget ran out before the evidence arrived, which is a fact about the gold set rather than a clean bill of health — and a gate that treats "we could not tell" as a pass has quietly stopped gating.

As a GitHub Action

- uses: jinwovo/rag-eval-gate@v1
  with:
    gold-file: eval/gold.jsonl
    url-template: 'http://localhost:8080/api/search?q={query}&mode={mode}'
    gate-policy: sequential
    min-mrr10: "0.85"

The step summary gets the interval table, the significance verdicts, what the gold set can resolve, and a ✅/❌.

What comes out

queries=300  depth=10  intervals=95%

mode                      recall@5             recall@10                mrr@10               ndcg@10
--------------------------------------------------------------------------------------------------
bm25              0.700 [0.35,0.93]    0.800 [0.44,0.97]    0.670 [0.35,0.90]    0.702 [0.38,0.90]
hybrid            1.000 [0.69,1.00]    1.000 [0.69,1.00]    0.950 [0.70,1.00]    0.963 [0.78,1.00]

paired randomization test vs bm25 (Holm-corrected within each metric)
  recall@5   hybrid    delta=+0.300 p=0.2500 holm=0.2500  unresolvable (3/10 queries differ; floor p = 0.250)
  mrr@10     hybrid    delta=+0.280 p=0.1250 holm=0.2500  unresolvable (4/10 queries differ; floor p = 0.125)

resolution of this gold set (hybrid vs bm25, mrr@10):
  n = 10 queries, per-query difference sd = 0.388
  smallest detectable effect at 80% power: +0.344
  queries needed to resolve +0.020: 2,957

unresolvable is the word this tool exists to print. It is not a softer way of saying "not significant" — it means the queries that differ are too few for any outcome to reach p < 0.05, which is a property of your labels and no amount of rerunning will change it.

The statistics, and why each one

Every one is seeded, so a gate cannot flap because a resample drew differently.

Certified thresholds

rag-eval-gate calibrate gold.jsonl --alpha 0.10 --risk-alpha 0.05

Turns the two constants that decide what a RAG system costs and how often it can make things up into thresholds with finite-sample, distribution-free guarantees:

coverage   P(the passages sent to the LLM contain a relevant document)  >= 1 - alpha
risk       P( P(answering from context with no relevant document) <= risk-alpha ) >= 1 - delta

Measured on 800 queries against a stand-in cross-encoder: 5.0 passages against a fixed 9 carrying the identical promise — 44% fewer prompt tokens, because the budget moves to where it is needed (3.2 passages when the answer ranked first, 8.4 when it ranked fourth or lower). Three splits — one picks the softmax temperature, one fits the quantile, one scores both having informed neither — because choosing the temperature on the data the quantile is fitted on breaks exchangeability, and the resulting under-coverage is invisible in every in-sample number.

A certificate that abstains on everything is valid, useless, and labelled degenerate. A threshold search that stalls names the calibration size that would settle it.

The certificate above expires

The guarantee from calibrate holds while calibration and serving queries are draws from the same distribution. A live corpus stops satisfying that in a week — documents get ingested, query mix moves, someone swaps the embedding model — and the failure is silent: config unchanged, dashboards green, promise no longer true.

from rag_eval_gate import AdaptiveConformal

controller = AdaptiveConformal(alpha=0.10, gamma=0.05, window=600)
for score in calibration_scores:
    controller.scores.append(score)          # seed the window, no feedback yet

# serving
cutoff = controller.threshold()              # apply this
controller.observe(score)                    # then close the loop
print(controller.report())                   # offset, whether the bound held, compensating

The level is driven by realised miscoverage (Gibbs & Candès, NeurIPS 2021). Summing the update telescopes to an exact identity and the level provably cannot escape [-γ, 1+γ], so long-run coverage converges to target under no distributional assumption at all — not exchangeability, not stationarity, not even that the shift is random rather than adversarial. DtACI (JMLR 2024) runs several step sizes as experts if you would rather not pick γ.

200 streams, 3,000 queries each, promise 90%:

The middle column is the honest part. Against a plain location shift, rolling recalibration does essentially all the work by itself — across shifts of 0.00, 0.20 and 0.30 the widest excursion of the level from target is the same 0.100, so the shift never reaches the controller. If that is your only worry, recalibrate on recent traffic and skip this module. What the controller buys is the residual (the window is systematically off even when it works, 87.6% against a promised 90%) and the case no rolling quantile can track: when most candidates pile onto one tied top score, the window overcovers at 93.9% — not safety, just prompt tokens spent on sets larger than the guarantee needs.

Which is why the signal is compensating, not drifting. It stays False through a shift the window absorbs (0/60 streams) and fires on every saturating one (60/60). Naming it a drift detector would have been wrong, and the number that first looked like drift detection turned out to be an artefact of a test generator clamping scores at 1.0.

When only an LLM has judged your data

Your groundedness score is a judge model's opinion of the system it belongs to. Your relevance labels, at any real scale, were written by a model too. Both are cheap and neither is trustworthy on its own, and averaging thousands of them does not produce a 95% interval — it produces a very narrow interval around whatever the model believes.

ppi_mean implements prediction-powered inference (Angelopoulos et al., Science 2023, power-tuned as PPI++): hand-label a small sample, let the model predict everything, and subtract the bias measured on the labelled part.

from rag_eval_gate import ppi_mean

estimate = ppi_mean(hand_labels, judge_on_those, judge_on_everything_else)
print(estimate.statement())
# The judge reports 0.886. Measured against 50 hand labels it is optimistic by 0.081,
# so the true value is 0.805 [0.781, 0.829] — an interval worth about 474 hand labels,
# from 50.

Over 1,500 repetitions of the whole label-and-estimate cycle, 50 hand labels against 2,050 judge-scored items:

Validity never depends on the judge being good — the bias is measured, not assumed. And a useless judge costs nothing: λ is tuned to minimise variance, so noise gets λ ≈ 0 and the estimator falls back to the hand-label mean. That last row is the safety property, not a failure.

Why zero dependencies

This runs inside a CI job that should not need a resolver, a wheel build or a lockfile to gate a pull request. The machinery it needs — a regularized incomplete beta, a BCa bootstrap, a martingale — is a few hundred readable lines, and pip install retrieval-eval-gate installs exactly one thing. The test suite cross-checks the hand-rolled special functions against SciPy, and skips those checks when SciPy is not installed.

Tests

pip install -e ".[dev]" && pytest

220 tests. Against closed forms ((α/2)^(1/n) at k = n, 2^(1-k) for a uniform improvement, R's p.adjust worked example, the (n+1) conformal quantile correction), against an independent brute-force permutation reference written inside the test, and — for the guarantees — by simulating the whole calibrate-then-deploy cycle and counting how often each promise actually breaks.

Provenance

Extracted from jinwovo/recall, a hybrid-search and grounded-RAG system, after its own eval harness was pointed at its own README and three of the headline claims failed. The design notes live there as ADRs 0011–0014.

License

MIT.