verity

Measure retrieval, don't assume it. Deterministic metrics, and filtered vector search that actually returns k results.

Two problems

1. Filtered vector search silently returns fewer results than you asked for

Ask a vector store for the 10 nearest neighbours where lang = 'py' and most of them do this:

candidates = index.search(query, k=10)             # the filter is ignored here
return [c for c in candidates if c.lang == "py"]   # and applied here

If only 3 of the 10 nearest neighbours are Python, you get 3 results — not the 10 nearest Python documents. No error. The request looks successful. pgvector has these semantics, and so does the default HNSW path in several other stores.

It gets worse as the filter gets more selective, and it hides from the metric most people check: precision@10 over 3 results that happen to be correct scores 1.0 if you divide by what came back instead of by k.

2. Retrieval quality is measured by asking a language model

Every major RAG framework evaluates retrieval indirectly — an LLM judges whether the retrieved context looks relevant. That buys a judge with a 13.6% flip rate on repeated runs, to approximate quantities that are exact, free, and standard since the 1990s.

An LLM judge is the right tool for is this answer good. It is the wrong tool for did the retriever return the document I already know is relevant — that question has a ground truth, and comparing against ground truth doesn't require judgement.

Measured, not asserted

verity bench produces this table. Fixed seed, exhaustive oracle in the same process, so the numbers reproduce on your machine:

2,000 documents · 128 dimensions · k=10 · 50 queries · degree=16, ef_search=64 · seed 42. short is the fraction of queries that came back with fewer than 10 results.

Read the post row at 3% selectivity: 0.36 results out of 10, on 100% of queries. The correct answer has 10, and pre proves it does. That is a RAG pipeline answering from almost nothing while reporting success.

pushdown returns 10 of 10 with perfect recall at every selectivity. At 100% — where the filter matches everything, so there is nothing to get wrong — post and pushdown converge to the same 0.786, which is ordinary approximate-search behaviour and confirms the gap at lower selectivities comes from the filtering rather than the index.

Why pushdown works: push the predicate into the graph traversal. A candidate that fails the filter is still a useful stepping stone — its neighbours may pass — so it is traversed but not collected. That is the idea behind ACORN-style predicate-aware search, and it is the whole difference between "3 of 10" and "10 of 10".

Use it

pip install verity-retrieval    # the import name is `verity`

Vectors may be plain lists — an embedding API returns JSON, so that is usually what you have — or numpy arrays. Either way they are L2-normalised on the way into an index.

Prove your store has the bug. Compare its filtered results against an exhaustive oracle:

from verity import BruteForceIndex, GraphIndex, FilterMode, Record, recall_loss

oracle = BruteForceIndex(records)          # exhaustive, therefore correct
graph  = GraphIndex(records)

exact = oracle.search(query, k=10, predicate=is_python)
actual = your_store.search(query, k=10, filter={"lang": "py"})

recall, shortfall = recall_loss(actual, exact)
if shortfall:
    print(f"asked for 10, got {10 - shortfall}: your store post-filters")

Score a retriever against a golden set:

from verity import Query, evaluate

queries = [Query("q1", "how does auth work?", relevance={"auth.py": 3.0, "session.py": 1.0})]
ev = evaluate(my_retriever, queries, k=10)
print(ev.summary())
print(ev.worst(5))          # the queries to go and debug

Fuse hybrid results:

from verity import reciprocal_rank_fusion

fused = reciprocal_rank_fusion({"bm25": bm25_ids, "vector": vector_ids}, limit=10)
fused[0].contributions      # {'bm25': 1, 'vector': 3} — which retriever put it there

Design decisions

recall_at_k returns 0.0 when nothing is relevant, not 1.0. Some libraries return 1.0 ("we found all zero of them"), which lets an unlabelled query inflate an average into looking perfect. The harness counts and excludes those queries instead, and reports the count — a golden set that's 30% unlabelled is a fact about your evaluation, not about your retriever.

Shortfall is reported separately from recall. They are different failures needing different fixes: imperfect ranking is normal for an approximate index; returning fewer results than requested is a contract violation. Averaging them together hides the second inside the first.

precision_at_k divides by k, not by what came back. This is what makes the bug above visible instead of flattering.

RRF sums contributions; it doesn't take the best rank. A surprisingly common implementation does scores[doc] = max(scores[doc], 1/(k+rank)), which discards exactly the cross-retriever agreement RRF exists to capture. There's a test asserting the sum.

No reranker. The evidence is weaker than its popularity: published comparisons put BM25 alone at 0.662 nDCG@10 in ~0.1 ms against 0.671 with a cross-encoder at ~225 ms — a gain inside the confidence interval for roughly 2000× the latency. verity gives you the fusion and the measurement; budget the rerank yourself, on your own corpus.

One dependency: numpy. No vector database, no embedding provider, no LLM client. A judge model in the dependency tree would undercut the entire argument, and it means the test suite runs offline with no credentials.

A bug this project found in itself

The first benchmark run showed pushdown degrading badly at higher selectivity — 86% shortfall at 10% — and unfiltered recall of only 0.36. The graph was fine. The search was quitting early: I had conflated the result set with the search frontier and terminated on a visit budget derived from it.

Separating them, as HNSW does — a bounded result heap, with termination when the nearest unexplored candidate is worse than the worst result held — moved unfiltered recall from 0.360 → 0.780 at ef=64, and to 1.000 at degree=32, ef=128.

Worth recording for two reasons. It is exactly the failure this library exists to catch: without an exhaustive oracle to measure against, "recall 0.36" is indistinguishable from "this is just how ANN works". And the fix is in TestTraversalQuality, which asserts recall is monotonic in ef_search and reaches >0.9 — so it can't come back.

What this is not

• Not a production vector store. GraphIndex holds vectors in memory and doesn't persist. It is a real navigable small-world graph — one layer, exact neighbour construction — so the comparison isn't a strawman, but use it to prove your store has this bug, then fix your store.
• Not an answer-quality evaluator. verity measures retrieval. Whether the generated answer is good is a different question, and one where an LLM judge is appropriate.
• Not a RAG framework. No chunking, no embedding, no orchestration.

Development

uv venv && uv pip install -e ".[dev]"
uv run pytest              # 90% coverage floor, enforced
uv run mypy src/verity     # strict
uv run ruff check .
uv run verity bench        # reproduce the table above

86 tests, 98% coverage, mypy strict, doctests in CI, green on Python 3.11–3.13.

Correctness is checked three ways: unit tests per metric and mode; property-based tests (Hypothesis) asserting metrics stay in [0,1], recall is monotonic in k, and pushdown never under-returns when k matching documents exist; and the benchmark itself, whose shape is asserted — post-filtering must under-return and pushdown must not, or the suite fails.

Licence

MIT