legal-rag-verifier

Sentence-level verification for legal RAG, while the model generates.

legal-rag-verifier checks each sentence of an answer against the statutory provisions it was given, as the sentence is written. A deterministic claim check covers figures, dates, citations, instrument titles, modal verbs and qualifiers; an NLI head (DeBERTa-v3) covers the prose. A rejected sentence is rolled back and regenerated on the inference server's prefix cache, so only that sentence is thrown away. The retry can insert the provision the sentence was checked against at the cut point, which makes a small model's retry pass the check far more often.

Status: proof of concept. The mechanism holds on a production inference server. With the DeBERTa head as the auditor the loop did not improve answers. With an LLM judge in its place it did, on the tests run so far. The package ships the DeBERTa head; the LLM judge is in scripts/. The verdict states what holds, what does not and what is required first.

Built by Abdullah Memon at Memon Systems Ltd. Layer 4 of the Memon Systems hosted reference architecture, after legal-rag-router, which can serve as its abstention gate.

Install

pip install legal-rag-verifier            # claim check only: no runtime dependencies
pip install "legal-rag-verifier[nli]"     # + DeBERTa-v3 NLI head (torch, transformers)
pip install "legal-rag-verifier[sglang]"  # + client for an SGLang server (tokenizer only)
pip install "legal-rag-verifier[hf]"      # + in-process Hugging Face generation

Python 3.11 or later. English and United Kingdom legislation in 0.1.0.

Check a sentence

from legal_rag_verifier.premise import Premise
from legal_rag_verifier.verifier import Verifier

premise = Premise.from_text(
    "The limit of a compensatory award is the lower of £123,543 and 52 multiplied by a week's pay.",
    titles=("Employment Rights Act 1996",),
    citations=("s124",),
)
verifier = Verifier()  # Verifier(SentenceNLIVerifier("cross-encoder/nli-deberta-v3-base")) adds NLI

verdict = verifier.check_sentence(premise, "The compensatory award is capped at £85,000.")
verdict.verdict  # Verdict.ROLLBACK
verdict.reasons  # (Reason.UNGROUNDED_FIGURE,)
verdict.ungrounded  # ['£85,000']

check_text runs the same checks over a whole answer, for post-hoc use.

Generate with rollback

from legal_rag_verifier.backends.sglang import SGLangBackend
from legal_rag_verifier.engine import InFlightGenerator
from legal_rag_verifier.nli import SentenceNLIVerifier

backend = SGLangBackend.connect("http://127.0.0.1:30000", "Qwen/Qwen2.5-7B-Instruct")
generator = InFlightGenerator(
    backend,
    Verifier(SentenceNLIVerifier("cross-encoder/nli-deberta-v3-base")),
    steering="inject",
)
answer = generator.generate_verified("What is the cap on the compensatory award?", premise)
answer.text  # the verified answer
answer.trace  # per sentence: verdicts, retries, what was inserted, prefix-cache hits, timings

The engine sends one request per sentence. A rejected sentence is not resubmitted, so the next request reuses the verified prefix from the server's cache (SGLang's RadixAttention). Each sentence gets at most three retries and each answer at most eight; after that the engine writes a fixed refusal for that point and moves on. Retry modes:

• allow: constrain a rejected figure to the grounded values of the same type, then ban;
• ban: restart the sentence with its rejected first token banned;
• inject: insert the provision the sentence was checked against at the cut point (a new user turn; inject_template, which may restate the question with {query}) and regenerate. The provision stays in the context and never enters the answer.

The SGLang server needs --enable-custom-logit-processor for allow and ban. Full API: docs/API.md.

Evidence

Every figure below is measured. Figures: docs/measurements.md. Methods, fixed before each run: docs/GRID.md. Raw results: results/.

Detector, sealed held-out battery (346 rows, sealed before it ran once; claim check + nli-deberta-v3-base + enrichment): 80.0 % of planted errors caught (124/155), 7.9 % of correct sentences rolled back (15/191), 89.2 % rollback precision. Weakest: dropped qualifiers (6/18).

Rollback on Qwen 2.5 7B, SGLang, one NVIDIA L4: every retry reused the verified prefix from the cache. A retry request costs 117 ms in bf16 (69 ms in FP8), against 270 ms (170 ms) to re-prefill. Checking sentence by sentence released the first verified sentence after 3.8 s, against 8.1 s when the whole answer was checked first.

Inserting the provision at the cut point (48 rejected sentences retried from the same state): the retry passed the auditor 39 times with the provision inserted, against 19 with decode-level steering (exact McNemar p = 1.1e-5). The same text at the top of the prompt passed 17 of 46; a new turn without the provision, 14. With the provision inserted, the regenerated sentence became more likely in 46 of 46 states, the rejected sentence less likely in 45 of 46, and attention on provision text rose in 46 of 46.

Live output, DeBERTa head as the auditor: its rejections were mostly wrong: 1 of 12 by hand on software questions, 18 of 46 by a frontier model's labels on statutes. Graded by hand, the loop's answers were no better than one plain answer from the same model. Inserting the provision makes the retry pass the auditor; it does not make it better.

An LLM judge in the auditor's place (Qwen 27B at 4 bits as generator and judge on one SGLang server; the judge is the entailment scorer, in scripts/live_loop.py): right on 93.8 % and 90.1 % of the sentences of two hand-graded tests (354 and 151 sentences), against 68.7 % and 58.3 % for the DeBERTa head. Run live on 40 questions with the source in the prompt, 48 of 49 sentences were committed at the first try, none wrong, at 1.9 s per answer at the median. When the answer started from memory, 36 of 65 sentences were wrong without the loop and 3 of 50 with it, all three refusals.

Other judges on the same two tests: a 1B model trained on the 27B's labels, 89.0 % and 82.1 % at 0.06 s per call (adapters); Jev 1.13, 90.7 % and 88.1 %; the OpenAI Decisions API, 89.0 % and 86.8 %. None is within 3 points of the 27B on both tests.

See docs/VERDICT.md for the proof-of-concept verdict and its limits.

Reproduce the evidence

The repository, not the package, holds everything behind the figures: the batteries and their seals (batteries/), the per-provision enrichment (enrichment/), every answer, trace and judgement (results/) and the scripts that produced them (scripts/). The rollback-point tables and the placement and attention report re-derive from the committed results, with no GPU or API key:

git clone https://github.com/azterizm/legal-rag-verifier && cd legal-rag-verifier && uv sync
uv run python scripts/grid_inject.py --cells 4B 4B-inject 4B-inject-v2
uv run python scripts/replay_report.py

Re-scoring the grid (scripts/grid_score.py) and re-running the batteries also need the statute corpus, built with legal-rag-router and copied to data/; it is not redistributed here. Generation runs used Modal (scripts/modal_m4.py).

Development

uv sync
uv run ruff check && uv run ruff format --check && uv run mypy && uv run pytest

Progress and decisions: docs/ROADMAP.md. Contributing: CONTRIBUTING.md. Security: SECURITY.md.

Citation

If you use the verifier or its evaluation, cite it with CITATION.cff ("Cite this repository" on GitHub).

Consultancy

legal-rag-verifier is built and maintained by Abdullah Memon at Memon Systems Ltd, which designs and audits retrieval systems for legal and other regulated domains. For help putting verification into production or testing your own legal RAG, see engagements or write to abdullah@memonsystems.com.

Licence

AGPL-3.0-only (see LICENSE). Contains public sector information licensed under the Open Government Licence v3.0 (see NOTICE).