rag-ablations

Most RAG projects claim they improved retrieval. Almost none say what they improved it over.

This is a retrieval benchmark that always states its baseline. Every design choice (chunking, sparse vs dense vs hybrid, reranking) is measured against classical BM25 on public corpora with real relevance judgments, and every number in this README is regenerated by one command.

It runs on a laptop CPU in minutes, with no API keys and nothing to pay for, so anyone reading this can reproduce the table rather than take it on trust. CI re-runs the benchmark on every push and fails if a number moves, so the README cannot quietly drift away from the code.

Is the baseline itself correct?

That question decides whether anything else here means anything, so it is checked against published figures rather than asserted. The BEIR paper reports Anserini BM25 at 0.665 nDCG@10 on SciFact and 0.325 on NFCorpus. This independent implementation:

Corpus	Published Anserini BM25	This implementation
SciFact	0.665	0.6862
NFCorpus	0.325	0.3219

Within a point on both. The baseline is sound, so the comparisons below are worth reading.

Results

SciFact: 5,183 documents, 300 test queries, single-threaded CPU.

System	nDCG@10	Recall@100	ms/query
BM25 (stemmed), baseline	0.6862	0.9209	0.8
BM25 (no stemming)	0.6663	0.8859	0.5
Dense (MiniLM-L6)	0.6451	0.9250	18.1
Hybrid (BM25 + dense, RRF)	0.7153	0.9550	35.7
BM25 + cross-encoder rerank	0.6875	0.9209	7932.7
Hybrid + cross-encoder rerank	0.6874	0.9550	7670.6

Three findings, and two of them are negative:

Dense retrieval lost to BM25: 0.6451 against 0.6862, at 20× the query cost and five minutes of indexing against one second. This is not a broken implementation; it is the well-documented BEIR result that bi-encoders trained on MS MARCO degrade out of domain, and SciFact's scientific claims are firmly out of domain. Note the split signal: dense had higher recall@100 (0.9250) and worse nDCG@10, meaning it found the relevant documents but ranked them worse.

Hybrid was the only real win: 0.7153, beating the baseline by 0.029 nDCG@10 and lifting recall@100 from 0.9209 to 0.9550. That the fusion beats both of its inputs is the point: the two retrievers fail on different queries, which is exactly the case RRF exploits, and the recall lift is evidence they are finding different documents rather than agreeing.

The cross-encoder reranker was not worth it. On top of BM25 it moved nDCG@10 by 0.0013, a tie. On top of hybrid it lost 0.028, dragging the best system back down to baseline. And it cost about 7.9 seconds per query against 36 milliseconds, roughly 200× slower, because a cross-encoder scores 100 query-document pairs per search instead of one dot product.

That last result is the one worth dwelling on. Reranking is close to a default recommendation in RAG write-ups, and here it was neutral at best, harmful on the strongest pipeline, and catastrophic for latency. The mechanism is visible in the recall column: reranking cannot change recall@100 when it reranks exactly the top 100, so all it can do is reshuffle, and an MS MARCO cross-encoder reshuffling out-of-domain scientific claims reshuffled them slightly wrong.

None of this generalises to your corpus. That is the argument for measuring rather than adopting.

Does chunking help?

Chunking is the RAG design choice most often decided by copying chunk_size=512, overlap=50 from a tutorial. Held against a fixed BM25 retriever, on these corpora, it does not help:

SciFact:

Chunking	nDCG@10	Recall@100
Whole document (control)	0.6862	0.9209
128-word windows, 32 overlap	0.6794	0.9126
3-sentence windows, stride 2	0.6578	0.9016
64-word windows, 16 overlap	0.6500	0.9076

NFCorpus:

Chunking	nDCG@10	Recall@100
Whole document (control)	0.3223	0.2464
128-word windows, 32 overlap	0.3164	0.2413
3-sentence windows, stride 2	0.3085	0.2429
64-word windows, 16 overlap	0.3076	0.2415

Smaller chunks were monotonically worse, and the popular default was worse than doing nothing. That is not a general law; it is what these corpora say. Both are built from abstracts and short articles that already sit near a single topic, so splitting them mostly dilutes term statistics and splits evidence across passages that then compete with each other. On long, multi-topic documents the result would likely invert. The point of measuring is that you do not have to guess which case you are in.

Note that chunking also costs query time: pooling chunk scores back to documents roughly doubled latency here for no gain.

Run it

pip install rag-ablations

python -m rag_ablations.benchmark --dataset scifact --systems sparse     # baseline, no model downloads
python -m rag_ablations.benchmark --dataset scifact --systems chunking   # chunking ablation
python -m rag_ablations.benchmark --dataset scifact --systems all        # adds dense + reranking

The corpus downloads itself on first use (2.8 MB) and is cached in data/, alongside the results/ the run writes, both under whichever directory you run from. No keys, no accounts. --systems all needs the dense extra (pip install "rag-ablations[dense]") and downloads two small models from HuggingFace.

To work on it rather than run it, clone and pip install -e ".[dev]". In a checkout the same two directories sit next to src/, so --check compares against the committed results.

Add --check to any command to verify the stored results still reproduce instead of overwriting them. That is what CI runs.

The service

docker compose up
curl "http://localhost:8000/search?q=lithium&k=3"
curl "http://localhost:8000/health"

The API builds its retriever through the same factory the benchmark uses, so the published numbers describe what the service actually does, and /health reports the configuration that produced them. CI builds the image, starts it and issues a real query on every push.

Design decisions

Public corpora with relevance judgments, not a private document set. The alternative, indexing some documents and eyeballing whether the answers look good, cannot produce a number anyone can check. BEIR's SciFact and NFCorpus ship human judgments and published baselines, which is what makes these results falsifiable.

BM25 as the baseline, written from scratch. BM25 still beats dense retrieval on many out-of-domain corpora, so it is a real opponent rather than a straw man. It is implemented here instead of imported from rank_bm25 because it is the load-bearing number in the project: k1 and b are benchmark variables, and a baseline nobody can inspect is a baseline nobody should believe.

Metrics implemented directly rather than via pytrec_eval. That package needs a C build toolchain, which breaks "reproducible by anyone on a laptop". nDCG and recall are forty lines, and tests/test_metrics.py checks them against values computed by hand.

Two analyzers reported, not one. Published BM25 baselines use a stemming, stopword-removing Lucene analyzer. Reporting only the naive tokeniser would have understated the baseline by two points and quietly flattered everything measured against it; reporting only the stemmed one would hide how much of the score comes from the analyzer. Both are in the results.

RRF for hybrid retrieval, not score blending. BM25 scores are unbounded sums of idf terms while cosine similarities sit in [-1, 1], so blending them requires normalising two distributions whose shape changes per query, a tuning knob that quietly becomes a per-dataset fit. RRF uses rank position only and has a single constant, left at the published 60 rather than tuned against the test set. Choosing the method with fewer degrees of freedom is what keeps the comparison honest.

Recall@100 reported alongside nDCG@10. A reranker can only reorder what the first stage retrieved, so first-stage recall is a hard ceiling on the whole pipeline. Without it, a reranking result is uninterpretable, and one of the tests pins exactly that: a document the first stage missed cannot be recovered.

Chunk scores are max-pooled back to documents. Chunking changes the unit of retrieval, but the relevance judgments are written against whole documents. Scoring chunks directly would compare against the wrong ground truth, so every row stays scored against the same qrels.

Exhaustive cosine search, not an ANN index. At a few thousand documents, HNSW would add a dependency and an approximation error to a search that already takes milliseconds. Approximate indexes belong in a service at a corpus size that justifies them, not in a benchmark whose job is to isolate the effect of the model.

Local embeddings, no embedding API. Partly cost, mainly reproducibility: a hosted endpoint can change what it returns under a stable model name, and then the table describes a system nobody can reconstruct. The embedding cache is keyed on the corpus contents, so a chunking change cannot silently reuse stale vectors.

data/ and results/ are resolved, not assumed. The package was written inside a checkout, where those directories sit next to src/ and the committed results are what --check compares against. Installed from a wheel there is no checkout, and the same relative walk lands in the interpreter's library directory, so a plain install followed by a benchmark run would download a corpus into site-packages or fail where that is not writable. A checkout is now identified by its pyproject.toml; anywhere else the working directory is used, and RAG_ABLATIONS_HOME overrides both.

Dense retrieval is an optional install. pip install rag-ablations should not pull 2 GB of PyTorch on someone who wants the baseline and the metrics.

The benchmark runs in CI, not just the unit tests. The tests assert properties (that rare terms outweigh common ones, that term frequency saturates), so a refactor that quietly changed the analyzer or the idf formula would pass them all. --check re-runs the benchmark over 300 real queries and fails on drift.

certifi pinned for the corpus download. Python does not use the Windows trust store and does not chase AIA links for missing intermediate certificates, so the stdlib default fails on hosts that curl and every browser accept. Found on the first real download, not guessed at.

Limitations

• Two corpora, both scientific. SciFact and NFCorpus were chosen because they are small enough to run on a CPU and have published baselines. Conclusions here, especially the chunking result, should not be read as general.
• No hyperparameter search. k1, b and rrf_k sit at their published defaults. Tuning them against the test set would produce better numbers and worse evidence.
• Single run per system. These retrievers are deterministic, so there is no variance to report, but that also means no confidence intervals on the differences. Small gaps, the 0.0013 between BM25 and BM25-plus-reranking for instance, should be read as ties, not as wins.
• Dense, hybrid and reranking were run on SciFact only. The reranked configurations take about 40 minutes on a laptop CPU, so NFCorpus has the sparse and chunking ablations but not the dense ones yet.
• CI reproduces the sparse and chunking results, not the dense ones. Gating --systems all would put two model downloads and a 40-minute cross-encoder run on every push. The rows that CI cannot re-verify are the rows whose numbers you should trust least.

License

MIT