QuantGuard

Deterministic, unit-aware structured-claim verification for RAG systems.

Verify what can be computed. Escalate only what requires semantic reasoning.

A RAG system can retrieve the correct evidence and still generate an incorrect quantitative claim — a wrong number, the right number attached to the wrong entity, a number for the wrong year, a citation to the wrong clause. Asking an LLM "is this answer correct?" is slow, costs a model call, and is often worse than useless at exactly this kind of check — LLM judges are notoriously bad at telling that 37% contradicts 33%, or that clause 5.3.5.4 isn't clause 5.3.5.5.

QuantGuard checks these claims mathematically instead, with zero LLM calls in the common case:

from quantguard import QuantitativeGuard

guard = QuantitativeGuard(tolerance=0.05)

result = guard.verify(
    generated_text="Latency was reduced to 18ms per 3GPP TS 38.331 Section 5.3.5.4.",
    source_chunks=["Latency achieved was 12.4ms (3GPP TS 38.331 Section 5.3.5.4)."],
)

result.is_valid          # False
result.status_counts     # {"CONTRADICTED": 1, "VERIFIED": 1}
result.to_dict()          # JSON-serializable, with a full trace per claim

Installation

pip install rag-quantguard

The core has zero required dependencies. The import name is quantguard (the PyPI distribution is named rag-quantguard because quantguard was already taken), so usage always looks like from quantguard import QuantitativeGuard, regardless of how it was installed. Optional integrations (LangChain, LlamaIndex, OpenTelemetry, an NLI-based fallback) are pulled in via extras -- see v0.2: everything above the deterministic core below.

To work on QuantGuard itself instead of just using it:

git clone <this-repo>
cd rag-quantguard
pip install -e ".[dev]"   # editable install + pytest
pytest                     # run the test suite
python benchmarks/run_benchmark.py

What it verifies

Claim type	Example	Handles
Numbers	Latency was 12ms	exact + rounding-aware tolerance
Percentages	94.3% accuracy	as a first-class unit
Unit-aware values	5 km vs 5000 m	normalizes across compatible units
Ranges	between 10 and 20ms	inside/outside checks
Bounds	under 50ms, at least 95%	inequality checks
Arithmetic totals	484 tasks vs 262+197+22+3	subset-sum search over evidence
Derived percentages	54.1% vs 262/484	part/whole search over evidence
Technical references	3GPP TS 38.331 §5.3.5.4	organization/document/section matching

The anti-bug principle

QuantGuard never concludes a claim is true merely because the same number exists somewhere in the source:

guard.verify(
    "User A paid $20.",
    ["User A paid $10.", "User B paid $20."],
)
# CONTRADICTED -- User A actually paid $10; $20 belongs to User B.

guard.verify(
    "Latency in 2024 was 12ms.",
    ["2024 value = 25ms.", "2025 value = 12ms."],
)
# CONTRADICTED -- 12ms is real, but it's 2025's value, not 2024's.

This works via lightweight entity/context binding and temporal-context matching during evidence ranking (quantguard/evidence.py), not just numeric proximity.

Benchmark

benchmarks/run_benchmark.py compares QuantGuard against a naive baseline that only checks whether a claim's numeric substring appears anywhere in the source text (the failure mode this library exists to fix), across 22 cases spanning every claim type above:

QuantGuard accuracy : 22/22 = 100.0%
Naive accuracy      : 14/22 = 63.6%
QuantGuard p50 latency : ~0.2 ms
API cost             : $0 (no LLM call in the deterministic path)

No LLM-judge comparison is included: this was built without access to a hosted model to call as a judge, and only measured numbers are published here, per the project's own "compute first" philosophy — an estimated comparison would undercut the point.

Result taxonomy

Every claim resolves to one of six statuses, never a bare True/False:

• VERIFIED — exact match (within representation/rounding precision)
• APPROXIMATE — within configured tolerance, not exact
• CONTRADICTED — evidence exists and disagrees
• UNSUPPORTED — no relevant evidence found
• AMBIGUOUS — multiple evidence candidates disagree and neither is clearly preferred
• NOT_APPLICABLE — claim kind has no deterministic strategy (reserved for a future semantic/NLI fallback stage; not reachable in v0.1, since extraction only produces structured, computable claims)

Every VerificationResult carries a trace: list[str] explaining the decision step by step, and the whole result is JSON-serializable via .to_dict().

Design choices worth knowing about

• Two tolerances, not one. verification_tolerance (default 5%) is for noisy real-world measurements. arithmetic_tolerance (default 0.5%) is for exact arithmetic identities (a claimed total, a claimed part/whole percentage) — these should only tolerate floating-point/display rounding, not several percent of slack. Conflating the two would let a claim like "500 tasks" pass against a true total of 484 (a 3.3% gap) just because it's under a 5% measurement tolerance, even though 500 is simply wrong, not a rounding of 484.
• GB and GiB are different dimensions, on purpose. data_decimal (KB/MB/GB/TB, 1000-based) and data_binary (KiB/MiB/GiB/TiB, 1024-based) are never silently reconciled — the units API treats them as incompatible, matching the design principle against conflating the two conventions.
• The arithmetic/derived-value rescue only reports a contradiction when there's a confirmed anchor to a relevant chunk. When no directly-typed evidence exists at all (an UNSUPPORTED claim), the rescue only upgrades the result to a positive match (VERIFIED/APPROXIMATE) — it will not report a CONTRADICTED computed from an unconfirmed chunk, because that risks citing an unrelated chunk's numbers as if they were relevant. This trades recall for precision by design: a real gap here (distinguishing "this chunk happens to be on-topic" from "this chunk is coincidentally numeric") is exactly the kind of judgment call the optional NLI/LLM fallback stage exists for, not something the deterministic engine should guess at.
• Entity binding is a lightweight heuristic, not a parser. Subject/context binding looks at nearby words in the same sentence, with a case-sensitivity rule so single-letter entity labels ("User A") don't collide with the article "a" once lowercased — a real bug this implementation caught and fixed in its own dev process (see quantguard/claims.py). A stronger dependency-parse-based resolver (spaCy, kept optional) is the natural v0.2 upgrade.

v0.2: everything above the deterministic core

The core (extraction, evidence, verification, arithmetic) has zero required dependencies. Everything below is opt-in via extras (pip install "rag-quantguard[extra]") and lazily imported, so installing the core never pulls in a web framework or an LLM SDK you don't use.

Auto-correction

result = guard.verify("Latency was 18ms, and there were 500 tasks in total.", sources)
patched_text, patches = result.auto_correct()
# "Latency was 12.4ms, and there were 484 tasks in total."

Only proposes a patch when there's exactly one clear evidence-backed replacement -- never for AMBIGUOUS results, since picking between two equally-plausible values would be exactly the kind of guess this library exists to avoid. Also available as result.patches (list) and result.patched_text (str) directly on GuardResult.

CLI

pip install -e .   # registers the `quantguard` command
quantguard verify answer.txt --sources sources.json [--json] [--verbose] [--fail-on-invalid]
quantguard benchmark dataset.json

Stdlib-only (argparse) -- the CLI doesn't add a dependency the core doesn't already avoid.

NLI/LLM fallback for non-quantitative claims

Structured claims (numbers, units, references) are handled deterministically. A sentence like "the new architecture improves reliability" isn't computable at all -- fallback_handler routes exactly these (and only these) sentences to a pluggable handler:

from quantguard.fallback.base import LexicalOverlapFallback

guard = QuantitativeGuard(fallback_handler=LexicalOverlapFallback())
result = guard.verify(
    "Latency was 12.4ms, and this proves the system is now much more reliable.",
    ["Latency achieved was 12.4ms.", "The new architecture improves reliability."],
)

LexicalOverlapFallback is a real, working, dependency-free default -- word-overlap scoring plus a negation-mismatch check -- but it's a heuristic, not a real entailment model, and its confidence scores are capped accordingly. For production use, swap in a real model via the same protocol (quantguard/fallback/nli.py has documented adapter examples for a HuggingFace NLI pipeline or an arbitrary LLM call). Without a fallback_handler configured, semantic sentences are simply outside what QuantGuard checks, exactly as in v0.1.

Streaming verification

from quantguard.streaming.verifier import verify_stream

for event in verify_stream(guard, token_stream, source_chunks):
    print(event.result.claim.raw_text, event.result.status.value)

Finalizes a claim only once it's stable across an iteration and has a settle margin of subsequent text -- a number built up token-by-token ("1", "8", "ms") resolves to one claim ("18ms"), not three premature ones. Flushes any pending claim when the stream ends.

LangChain

from quantguard.integrations.langchain import QuantGuardRunnable

verified_chain = rag_chain | QuantGuardRunnable(guard)  # expects {"answer", "context"}
output = verified_chain.invoke({"input": "..."})
output["quantguard_result"].is_valid

Composes directly onto a chain via LCEL's |; extracts Document.page_content automatically. verify_langchain_output(guard, chain_output) is available for a one-off check without permanently wiring it into the chain.

LlamaIndex

from quantguard.integrations.llamaindex import verify_response, make_verifying_query_engine

result = verify_response(guard, query_engine.query("..."))
# or, to attach it automatically on every call:
verified_engine = make_verifying_query_engine(query_engine, guard)
response = verified_engine.query("...")
response.quantguard_result.is_valid

OpenTelemetry

from quantguard.integrations.opentelemetry import InstrumentedGuard

guard = InstrumentedGuard(QuantitativeGuard())
result = guard.verify(answer, sources)  # same call, now traced

Emits one span (quantguard.verify) per call with claim-count/validity/per-status attributes, a span event for each CONTRADICTED/AMBIGUOUS/UNSUPPORTED claim (kept off VERIFIED/APPROXIMATE claims so event volume doesn't scale with every correct claim), and a quantguard.claims.verified counter metric labeled by status.

Project layout

quantguard/
├── quantguard/
│   ├── models.py            # Claim, Evidence, VerificationResult, Span, Status
│   ├── units.py               # Unit tables + dimension-aware normalization
│   ├── extraction.py           # Regex-based number/range/bound/reference extraction
│   ├── claims.py                # Claim construction + entity/context/temporal binding
│   ├── evidence.py                # Evidence extraction + candidate ranking
│   ├── verification.py             # Exact/tolerance/bounds/range/reference comparison
│   ├── arithmetic.py                # Subset-sum + derived-percentage verification
│   ├── guard.py                       # QuantitativeGuard: the public orchestrator
│   ├── cli.py                          # `quantguard verify` / `quantguard benchmark`
│   ├── correction/patches.py            # Auto-correction (Patch, propose_patch)
│   ├── fallback/                          # Optional NLI/LLM stage
│   │   ├── base.py                          # FallbackHandler protocol + LexicalOverlapFallback
│   │   └── nli.py                            # transformers/LLM adapter examples
│   ├── streaming/verifier.py                   # verify_stream()
│   └── integrations/                             # Optional framework glue
│       ├── langchain.py
│       ├── llamaindex.py
│       └── opentelemetry.py
├── tests/                 # 80 tests, including regression tests for real bugs found
├── examples/basic_usage.py
├── benchmarks/             # dataset.json + run_benchmark.py (vs. naive baseline)
└── pyproject.toml

Install

pip install -e ".[dev]"              # core + test tooling only
pip install -e ".[all]"              # everything, for exercising every integration
pip install -e ".[langchain]"        # just one extra, e.g. for LangChain users
pytest
python examples/basic_usage.py
python benchmarks/run_benchmark.py
quantguard verify examples_answer.txt --sources examples_sources.json