rag-eval-gate

Retrieval evaluation that tells you what it can't resolve.

Recall@5 = 1.00 on 10 queries is really [0.69, 1.00]. That +0.30 lift over BM25 has a p-value floor of 0.25 — no outcome of that comparison could ever have been significant. Both of those are computable from the published table alone.

→ Check a number right now, nothing to install

Type in the figure and the query count. It returns the interval that figure supports and, for a claimed improvement, whether any outcome of that comparison could have reached significance. The arithmetic runs in the page; nothing is sent anywhere.

Or from a terminal:

pip install retrieval-eval-gate      # the PyPI name; `rag-eval-gate` there is someone else's

rag-eval-gate audit -n 10 -v 1.00 -b 0.70     # audit a table you are reading
rag-eval-gate power gold.jsonl                # what your own gold set can resolve

Zero dependencies. Works with any retriever in any language. Runs as a GitHub Action. The command is rag-eval-gate; the distribution is retrieval-eval-gate because the shorter name on PyPI belongs to an unrelated project. Both spellings work as the command.

Three things your eval is probably not telling you

A perfect score is not a perfect score. Ten queries, one relevant document each, Recall@5 = 1.00 — that is ten successes out of ten, and the exact binomial interval bottoms out at (α/2)^(1/n) = 0.69. The headline is 31 points wide.

Some comparisons cannot be won. Your hybrid retriever beats BM25 by +0.30 Recall@5. On ten queries that is three queries flipping, seven tying — and a paired randomization test on three differing pairs has a smallest attainable two-sided p of 2/2³ = 0.25. Not "not significant": unable to be, whatever the result.

Your interval breaks when you look at it. A 95% interval promises that one look at a finished sample contains the truth 95% of the time. Watching a CI log is not that. Over 800 simulated evaluation streams, inspection starting only at n = 30 where the normal approximation is defensible:

Runs where the true mean escapes a nominal 95% interval at some point while it is watched. The fixed-N column grows with stream length — the longer you look, the more chances to be wrong. That is the law of the iterated logarithm, and it is why the right-hand column exists.

Audit a table in five seconds

No gold set, no retriever, no data — the reported means and the query count are enough:

$ rag-eval-gate audit -n 10 -m recall@5 -v 1.00 -b 0.70

recall@5 = 1.000 on 10 queries
  that is 10/10 successes

  95% interval   [0.692, 1.000]   width 0.308   (exact binomial)
  The reported figure is 31 points wide. Quoting it to two decimal places implies a
  precision 10 queries cannot deliver.

Claimed lift +0.300 over 0.700
  note: the two means determine it exactly: 3 of 10 queries differ

  smallest attainable p   0.2500

  UNRESOLVABLE. Under every possible overlap of the two systems' successes,
  a paired test on this many differing queries has a floor above 0.05, so no
  outcome of that comparison could have reached significance. This is a fact
  about the sample size, not about the systems.

It exits non-zero when the comparison could not have been significant, so it works as a check and not only as something to read. The interval is exact for a 0/1 metric; for a graded one the per-query spread is unknown, so it reports the widest interval the mean allows — a [0,1] variable with mean m has variance at most m(1−m).

The same thing runs in a browser at jinwovo.github.io/rag-eval-gate, with a shareable link for the result.

Quickstart

pip install retrieval-eval-gate

# 1. What can this gold set even measure?  (no retriever needed)
rag-eval-gate power gold.jsonl

# 2. Score it, with intervals and significance
rag-eval-gate eval gold.jsonl --modes bm25,vector,hybrid

# 3. Gate a pull request
rag-eval-gate eval gold.jsonl --gate --min-mrr10 0.85

A gold set is JSONL:

{"query": "how do I rotate a signing key", "relevant_doc_ids": ["kms-rotation"]}

…or the TREC qrels you already have (--gold qrels/test.tsv --queries-file queries.jsonl).

Works with whatever you built

Three ways in, so the language and shape of your retriever are irrelevant.

HTTP — a URL template and a small field map. No assumption about the route or schema:

rag-eval-gate eval gold.jsonl \
  --url-template 'https://search.internal/v2/query?text={query}&strategy={mode}' \
  --results-path 'data.hits' --id-field 'document_id' --score-field 'relevance'

A command — anything that prints JSON to stdout. Go, Rust, a notebook, a shell script:

rag-eval-gate eval gold.jsonl \
  --backend command --command './retrieve --q {query} --mode {mode} --json'

A TREC run file — rank the corpus once, offline, on whatever hardware; evaluate as often as you like with nothing running:

rag-eval-gate eval qrels/test.tsv --backend run-file --run-file runs/bm25.trec

Four gate policies

rag-eval-gate eval gold.jsonl --gate --gate-policy sequential

regression is the sensitive one. An absolute threshold only notices a regression once the mean crosses a line someone guessed; a paired test compares the same queries before and after, so a real drop on two of them is caught while the mean still clears the line.

sequential is the cheap one. Because a confidence sequence is valid at every sample size, stopping as soon as the verdict is settled is a decision rather than a peek:

undecided fails. The budget ran out before the evidence arrived, which is a fact about the gold set rather than a clean bill of health — and a gate that treats "we could not tell" as a pass has quietly stopped gating.

As a GitHub Action

- uses: jinwovo/rag-eval-gate@v1
  with:
    gold-file: eval/gold.jsonl
    url-template: 'http://localhost:8080/api/search?q={query}&mode={mode}'
    gate-policy: sequential
    min-mrr10: "0.85"

The step summary gets the interval table, the significance verdicts, what the gold set can resolve, and a ✅/❌.

What comes out

queries=300  depth=10  intervals=95%

mode                      recall@5             recall@10                mrr@10               ndcg@10
--------------------------------------------------------------------------------------------------
bm25              0.700 [0.35,0.93]    0.800 [0.44,0.97]    0.670 [0.35,0.90]    0.702 [0.38,0.90]
hybrid            1.000 [0.69,1.00]    1.000 [0.69,1.00]    0.950 [0.70,1.00]    0.963 [0.78,1.00]

paired randomization test vs bm25 (Holm-corrected within each metric)
  recall@5   hybrid    delta=+0.300 p=0.2500 holm=0.2500  unresolvable (3/10 queries differ; floor p = 0.250)
  mrr@10     hybrid    delta=+0.280 p=0.1250 holm=0.2500  unresolvable (4/10 queries differ; floor p = 0.125)

resolution of this gold set (hybrid vs bm25, mrr@10):
  n = 10 queries, per-query difference sd = 0.388
  smallest detectable effect at 80% power: +0.344
  queries needed to resolve +0.020: 2,957

unresolvable is the word this tool exists to print. It is not a softer way of saying "not significant" — it means the queries that differ are too few for any outcome to reach p < 0.05, which is a property of your labels and no amount of rerunning will change it.

The statistics, and why each one

Every one is seeded, so a gate cannot flap because a resample drew differently.

Certified thresholds

rag-eval-gate calibrate gold.jsonl --alpha 0.10 --risk-alpha 0.05

Turns the two constants that decide what a RAG system costs and how often it can make things up into thresholds with finite-sample, distribution-free guarantees:

coverage   P(the passages sent to the LLM contain a relevant document)  >= 1 - alpha
risk       P( P(answering from context with no relevant document) <= risk-alpha ) >= 1 - delta

Measured on 800 queries against a stand-in cross-encoder: 5.0 passages against a fixed 9 carrying the identical promise — 44% fewer prompt tokens, because the budget moves to where it is needed (3.2 passages when the answer ranked first, 8.4 when it ranked fourth or lower). Three splits — one picks the softmax temperature, one fits the quantile, one scores both having informed neither — because choosing the temperature on the data the quantile is fitted on breaks exchangeability, and the resulting under-coverage is invisible in every in-sample number.

A certificate that abstains on everything is valid, useless, and labelled degenerate. A threshold search that stalls names the calibration size that would settle it.

When only an LLM has judged your data

Your groundedness score is a judge model's opinion of the system it belongs to. Your relevance labels, at any real scale, were written by a model too. Both are cheap and neither is trustworthy on its own, and averaging thousands of them does not produce a 95% interval — it produces a very narrow interval around whatever the model believes.

ppi_mean implements prediction-powered inference (Angelopoulos et al., Science 2023, power-tuned as PPI++): hand-label a small sample, let the model predict everything, and subtract the bias measured on the labelled part.

from rag_eval_gate import ppi_mean

estimate = ppi_mean(hand_labels, judge_on_those, judge_on_everything_else)
print(estimate.statement())
# The judge reports 0.886. Measured against 50 hand labels it is optimistic by 0.081,
# so the true value is 0.805 [0.781, 0.829] — an interval worth about 474 hand labels,
# from 50.

Over 1,500 repetitions of the whole label-and-estimate cycle, 50 hand labels against 2,050 judge-scored items:

Validity never depends on the judge being good — the bias is measured, not assumed. And a useless judge costs nothing: λ is tuned to minimise variance, so noise gets λ ≈ 0 and the estimator falls back to the hand-label mean. That last row is the safety property, not a failure.

Why zero dependencies

This runs inside a CI job that should not need a resolver, a wheel build or a lockfile to gate a pull request. The machinery it needs — a regularized incomplete beta, a BCa bootstrap, a martingale — is a few hundred readable lines, and pip install retrieval-eval-gate installs exactly one thing. The test suite cross-checks the hand-rolled special functions against SciPy, and skips those checks when SciPy is not installed.

Tests

pip install -e ".[dev]" && pytest

200 tests. Against closed forms ((α/2)^(1/n) at k = n, 2^(1-k) for a uniform improvement, R's p.adjust worked example, the (n+1) conformal quantile correction), against an independent brute-force permutation reference written inside the test, and — for the guarantees — by simulating the whole calibrate-then-deploy cycle and counting how often each promise actually breaks.

Provenance

Extracted from jinwovo/recall, a hybrid-search and grounded-RAG system, after its own eval harness was pointed at its own README and three of the headline claims failed. The design notes live there as ADRs 0011–0014.

License

MIT.