Groundlens: a proofreader for RAG answers

Groundlens is a proofreader for what your model writes. It marks the words your sources don't back — and shows you what each one should have said.

QUESTION    What is the invoice total?
SOURCE      ...the total amount due is 10,000 dollars, payable within 30 days...
ANSWER      The invoice total is 1,000 dollars, due in 30 days.

GROUNDLENS  1,000   nothing supports this.   Closest in invoice.pdf#p1: '10,000'

It never tells you the answer is wrong. It tells you which word to look at, and which document to open. Thirty seconds of human attention instead of five minutes.

Install

pip install groundlens              # zero runtime dependencies. Not numpy, not torch
pip install "groundlens[encoder]"   # + the reference sentence encoder
pip install "groundlens[mcp]"       # + the MCP connector

The core install pulls nothing, and a CI job fails the build if that ever changes. The previous version installed roughly two gigabytes of deep learning stack before you had done anything.

Quick start

from groundlens import proofread, SentenceTransformerEncoder

answer = "The invoice total is 4.75% payable within 45 days."
sources = [("policy.pdf#p3", "The rate stated in the policy is 3.90% and the term is 30 days.")]

marks = proofread(answer, sources, encoder=SentenceTransformerEncoder(), k=2)

print(marks.report())
#  4.75%   support 0.00    nearest in policy.pdf#p3: '3.90%'
#  45      support 0.00    nearest in policy.pdf#p3: '30'

Every mark carries its receipt:

for anchor in marks.weakest:
    anchor.text            # '4.75%'          the word in the answer
    anchor.span            # (21, 26)         where it sits
    anchor.kind            # 'numeral'        checked by arithmetic, not meaning
    anchor.support         # 0.0              absent from the sources
    anchor.evidence_id     # 'policy.pdf#p3'  which document to open
    anchor.evidence_text   # '3.90%'          what it should have matched

From the shell:

groundlens read --answer answer.txt --context policy.pdf#p3=policy.txt

How it works

Words are anchored by meaning. A word's support is the highest cosine similarity it reaches against any word of the sources, using a frozen off-the-shelf encoder — the same kind your retrieval already uses.

Numbers are anchored by arithmetic. The numeral is parsed to a value with formatting normalised — 10,000, 10000, $10,000, 10 000 and (under a declared locale) 10.000 are one number — then checked against every value in the sources. Support is exactly 1.0 or exactly 0.0. Similarity is not allowed to vote.

And we report the floor, not the average. Every token-similarity metric aggregates by the mean, and the mean is where single-token errors go to die.

Ten is not a hundred

A retrieved document says the total due is 10,000 dollars. The answer says 1,000 dollars. A human catches that instantly, without a finance degree.

Embedding similarity does not. Cosine between the right answer and the wrong one is about 0.99 — the error dissolves into the vector the way a drop of ink dissolves in a pool. An LLM judge does not either: it reads for plausibility, and "the total is 1,000 dollars" is a perfectly plausible sentence about an invoice. A trained span detector does not, because single-digit substitutions are rare in its training labels.

Sentence encoders organise text by vocabulary, topic and structure. Never by truth. A wrong number inside a correct sentence is, to a paraphrase-collapsing encoder, very nearly a paraphrase.

On that invoice, the mean support of the wrong answer is 0.79 — which looks fine. The weakest anchor is 0.00 — which is a mark in the margin.

Why there is no threshold

We measured nine detectors across five public benchmarks — two published encoder models, an NLI cross-encoder, an LLM judge, and this one — at the operating point production actually runs at: false-positive rate at 95% hallucination recall.

Forty-five cells across the full grid. The best is 0.65. A random detector sits at 0.95. One method ranks best of all by AUROC and flags 99% of correct answers at the operating point. Nobody is in the usable corner — including us.

So proofread() returns no verdict and the library ships no default cut. If it did, someone would deploy it and be escalating two thirds of their clean traffic within a week. That is not a limitation of this library; it is the finding, and marks-not-verdicts is what you build once you take it seriously.

If you need a threshold, fit it on your own labelled traffic and read what it costs you:

from groundlens import calibrate

point = calibrate(labelled, target_recall=0.95)
print(point.threshold, point.fpr, point.fpr_ci95)   # read the fpr first

It refuses to run on fewer than 200 labelled examples, because below that a 95%-recall threshold is estimated from a handful of points.

Limitations

• It cannot verify computed values — "revenue tripled" against a source saying "revenue went from 5M to 15M".
• It cannot check reasoning. That belongs to entailment models.
• It inherits your retrieval. If the passage is wrong, so is the answer's grounding.
• Segmentation assumes space-delimited scripts, and warns rather than pretending when the text is largely CJK or Thai.

Reproducibility

• The numeral channel is exact. Decimal comparison, fixed arithmetic context, locale from an argument and never from LC_ALL. Byte-for-byte identical on any machine — CI proves it on ten OS × Python combinations under PYTHONHASHSEED=random and a Turkish locale.

• The lexical channel is a float32 cosine from a pinned encoder revision — not a model name, because a silent re-upload would change every number you ever published. It reproduces to 1e-6 across platforms and the ordering of the weakest anchors is stable. It is not bit-identical between x86 and Apple Silicon, and we make no claim that it is.

• marks.sha256 covers the structure and the numeral supports exactly, and rounds lexical supports to six decimals. Reproducing the hash reproduces the finding, not the last bits of the arithmetic.