claimcheck

Domain-tuned retrieval + zero-LLM claim verification, in one pipeline.

claimcheck glues two siblings — adaptmem (domain-adapted bi-encoder retrieval) and halluguard (reverse-RAG hallucination detection) — into a single API:

from claimcheck import Pipeline

pipeline = Pipeline.from_corpus(
    documents=["..."],
    labelled_queries=[{"query": "...", "relevant_ids": [...]}],
    train=True,        # fine-tune the retriever on the labelled set
    enable_nli=True,   # add NLI verification on top of cosine retrieval
)

verdict = pipeline.check(
    answer="The user prefers PostgreSQL because it has better JSON support.",
    question="What database does the user prefer?",
)

print(verdict.trust_score)         # 0.84
print(verdict.flagged_claims)       # ["...because it has better JSON support"]

What it is

A thin orchestration layer over the two siblings:

1. adaptmem trains a domain-adapted bi-encoder on your corpus + labelled queries.
2. halluguard wraps the trained encoder in a Guard with NLI verification, surfaces a per-claim and per-response trust score.

The same Pipeline object can be saved + reloaded as a unit, so a downstream service has one model directory to manage.

Why one package

Adaptmem and halluguard are independently useful:

• adaptmem alone is a retrieval-quality lift (any domain).
• halluguard alone is a verification layer (any encoder).

But the most common deployment shape pairs them — domain-tuned retrieval for the cosine gate, claim-level NLI on top. claimcheck saves you the wiring.

What it is NOT

• Not a wrapper around any LLM. Both siblings are explicitly LLM-free.
• Not a vector database. Bring your own; claimcheck is the encoder + verifier layer.
• Not a replacement for either sibling. If you only need adaptmem (no verification) or only halluguard (with a generic encoder), use them directly.

Daemon mode (Pipeline.from_daemon)

For deployments where you'd rather not load a SentenceTransformer in every Python process (claimcheck + halluguard + a third service each paying the same model cost), point claimcheck at a long-lived adaptmem serve process:

from claimcheck import Pipeline

# Daemon must be running: `adaptmem serve --port 7800`
pipeline = Pipeline.from_daemon(
    documents=[...],
    daemon_url="http://127.0.0.1:7800",
    enable_nli=True,   # NLI verifier still runs in-process
)
verdict = pipeline.check("an answer", question="...")

The encoder hop crosses HTTP; cosine search and NLI verification stay local. pipeline.save() is not supported for daemon-backed pipelines (the model lives in the daemon). Pipeline.from_daemon calls /healthz first so a misconfigured URL fails loudly at construction time, not deep inside the first .check().

How it compares to LLM-as-judge tools

The closest commercial / open-source category is "LLM-as-judge" — a separate large-model call grades each claim. Claimcheck is the no-LLM-judge branch: a deterministic NLI cross-encoder + retrieval-augmented gate. The tradeoffs are real and shape what you should use it for.

When claimcheck wins:

• High-throughput middleware where per-claim cost matters (every chatbot turn checked).
• Privacy-bound deployments (medical, legal, internal tools) where claims can't leave the perimeter.
• Domain-specific RAG where a tuned retriever beats a generic LLM judge that doesn't know your jargon.
• Streaming UX where users see the verdict as the LLM types.

When LLM-as-judge wins:

• Open-ended quality assessment ("is this answer helpful, safe, polite?") that isn't really a hallucination check.
• Few-shot domains with no labelled training queries to fine-tune the retriever.
• One-off audits where a $0.05 model call is cheaper than building infrastructure.

The two are complementary, not exclusive. A reasonable production stack runs claimcheck in-line on every response (cheap, deterministic, blocks the worst), and an LLM judge in a sampled audit (expensive, broader, catches subtler issues).

Status

v0.1.1 shipped. Public API decided (Pipeline.from_corpus, check, check_stream, check(profile=True), save/load), 6 unit tests passing, mypy --strict clean, CI matrix on 3.10/3.11/3.12. The two siblings (adaptmem v0.4-shipped, halluguard v0.2-ext-shipped) are mature enough to compose; this repo just wires them.

Pre-PyPI: install via local editable until siblings publish.

pip install -e ../adaptmem ../halluguard ../claimcheck

License

MIT.