ClaimLens

Claim-level evaluation for LLM outputs: decompose text into atomic claims, then verify every claim against a reference.

Instead of asking a judge LLM "rate this answer 1-10", ClaimLens extracts atomic (subject, predicate, object) claims from a response with an extractor LLM, then classifies each claim against a reference with a checker LLM: Entailment, Contradiction, or Neutral. Every score the toolkit produces is an aggregate over per-claim verdicts you can read, audit, and disagree with.

Commands

Command	What it does
ragcheck	RAGChecker-style RAG evaluation: 2 extractions + 4 checking directions, one self-contained JSON report (precision, recall, F1, faithfulness, and claim-level detail)
faithcheck	Faithfulness checking without ground truth: response claims vs the retrieved context. Works on live traffic.
refcheck	Classic reference checking: extraction + checking in one pass
extract / check / atomize	The individual building blocks, runnable standalone — walked through in order under examples/
eval extractor / eval checker	Meta-evaluation: measure the extractor and checker themselves against labeled data. Run eval checker first — eval extractor uses a checker, so an unqualified one makes its numbers partial

Why this instead of a single judge score

• Auditable: every metric decomposes into claim-level verdicts with explanations. You can see exactly which fact failed and why.
• Variance is first-class: LLM-based metrics are stochastic. Pass --runs N to ragcheck, faithcheck, eval extractor or eval checker and get mean +/- std [min, max] per metric across N full repetitions, plus each run's complete report. A single-run score overstates your certainty.
• The evaluator is itself evaluated: eval extractor and eval checker measure the measurement tool against ground-truth labels, so you know how much to trust the numbers before you compare systems with them.
• Honest failure handling: per-item errors (context too long, content policy, parse failure) are recorded as values next to the affected claim, never silently zeroed. A claim the checker failed to judge leaves both sides of the ratio rather than counting against the system under test, and a metric with an empty denominator is null, never 0.0. Abstentions ("I don't know") are first-class labeled outcomes: judged justified or unjustified, their cause apportioned between retriever and generator, and unanswerable questions ("gt_answer": "") expose unwarranted answers — while the metrics stay standard (a refusal is non-delivery; see docs/ragchecker.md#abstention).
• Any OpenAI-compatible endpoint: point the extractor and checker at different models, providers, or local servers independently.

Installation

Requires Python 3.12+. The project uses uv for dependency management — install it first if you don't have it:

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# macOS (Homebrew alternative)
brew install uv

# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Then clone and install into a virtual environment:

git clone https://github.com/Arthur-g-p/claimlens.git
cd claimlens
uv venv
uv pip install -e .

uv venv reads .python-version and downloads CPython 3.12 if it isn't already present. It does not touch your system Python.

Activate the environment before using the claimlens command:

source .venv/bin/activate     # macOS / Linux

.venv\Scripts\activate        # Windows

Activation is per-shell — you need it again in every new terminal. To skip it, prefix commands with uv run instead (uv run claimlens --help).

To also install the test dependencies (pytest, pytest-asyncio, coverage):

uv pip install -e ".[test]"

LiteLLM routing (a provider prefix on the model name, no base URL) is an optional extra. The default install talks to any OpenAI-compatible endpoint directly and stays small. To add the routing path:

uv pip install -e ".[litellm]"

From PyPI that is pip install "claimlens[litellm]".

Verify the install:

claimlens --help

Quick start

Copy the environment template and fill in your API keys:

cp .env-example .env

EXTRACTOR_API_KEY=...
CHECKER_API_KEY=...

Input is a JSON list of items. For ragcheck, each item needs response, gt_answer, and retrieved_context:

[
  {
    "question": "Who wrote Faust?",
    "response": "Faust was written by Goethe in the 19th century.",
    "gt_answer": "Johann Wolfgang von Goethe wrote Faust.",
    "retrieved_context": ["Goethe published Faust, Part One in 1808. ..."]
  }
]

The input file is a positional argument on every command — pass the path bare, with no flag in front of it.

You do not need to prepare data to start. examples/ ships a five-step walkthrough over one dataset, where each step needs a little more than the last, plus eval_data/ with human-labelled data for the eval commands.

Start at step 01 — decompose responses into claims:

claimlens extract examples/extract/kepler22b.json --extractor-model gpt-4o-mini

Run the full RAG pipeline (step 05), which needs gt_answer and retrieved_context as well:

claimlens ragcheck examples/ragcheck/kepler22b.json --extractor-model gpt-4o-mini --checker-model gpt-4o-mini

Repeat the whole experiment five times and report variance (5x LLM cost):

claimlens ragcheck examples/ragcheck/kepler22b.json --extractor-model gpt-4o-mini --checker-model gpt-4o-mini --runs 5

Check faithfulness with no ground truth at all (step 04):

claimlens faithcheck examples/faithcheck/kepler22b.json --extractor-model gpt-4o-mini --checker-model gpt-4o-mini

The model flags have short aliases (-e, -c, -m), but the long names are spelled the same way in every command, so those are the ones worth learning. examples/README.md lists which fields and which arguments each step needs.

The pipelines and evals (ragcheck, faithcheck, refcheck, eval extractor, eval checker) write two JSON files. The record holds everything: an _args block (what you asked for — every flag of the invocation), a _meta block (what the run turned out to be — counts, timings, derived keys), metrics where the command computes any, every count the console prints, and the complete per-item claim record — --runs N adds entries, it never reshapes. The _findings.json sibling is the review queue: only what went wrong, keyed by the console's own branch names, derived from the record. The building blocks (extract, check, atomize) instead emit the item list itself, enriched in place, so each one's output is the next one's input.

ragcheck writes a third file next to those two: {report_stem}.html, a self-contained viewer of the same record. One screen for the run — the main metrics with their spread over --runs, then every item as a row of dots, one per response claim, lime when a chunk grounds it — and one screen per item: GT claims, retrieved chunks and response claims in three columns with the verdicts drawn as lines between them, hover for the checker's explanation. It opens with a double-click, needs no server and loads nothing from the network. --no-html skips it. claimlens --help lists all commands and flags.

Use it from Python

The library is the same five verbs as the CLI, with the same keyword names as its flags. Keys come from EXTRACTOR_API_KEY and CHECKER_API_KEY in the environment or a .env, never as arguments.

import json, claimlens

items = json.load(open("data.json"))

record, findings = claimlens.ragcheck(items, extractor_model="gpt-4o-mini", checker_model="gpt-4o-mini")
record["metrics"]["faithfulness"]
findings["hallucination"]          # the review queue, the same as the _findings.json file

items = claimlens.extract(items, extractor_model="gpt-4o-mini")

ragcheck, faithcheck and refcheck return the record and the findings, the two documents the CLI writes. claimlens.render_html(record, findings) returns the viewer page for a ragcheck record as a string, the same page the CLI writes as {report_stem}.html. extract and check return the enriched item list. Extra keyword arguments go to the pipeline (concurrency, joint, extractor_base_url, runs, ...). Output is silent until you ask for it: claimlens.enable_logging() turns the console output on, and verbosity="compact" or "silent" per call sets how much.

Inside a running event loop (a Jupyter cell, an async server) the sync verbs refuse with a message that says so. Use the pipeline classes with await. It is the same code:

from claimlens.pipelines.ragchecker import RagCheckerPipeline

pipeline = RagCheckerPipeline(extractor_model="gpt-4o-mini", checker_model="gpt-4o-mini")
await pipeline.run(items)
record, findings = pipeline.last_report, pipeline.last_findings

One response, in real time. The async form first, because the caller is usually a server:

entry = await claimlens.acheck_faithfulness(response, chunks, extractor_model="gpt-4o-mini", checker_model="gpt-4o-mini")
entry = claimlens.check_faithfulness(response, chunks, extractor_model="gpt-4o-mini", checker_model="gpt-4o-mini")

The public API is what claimlens/__init__.py exports. Everything under claimlens.pipelines, .services and .workers is importable but may change between minor versions.

Using other providers and local models

The extractor and checker are configured independently, and there are two ways to reach a non-OpenAI endpoint.

Direct endpoint — pass a base URL with --extractor-base-api (or --checker-base-api). The request goes straight out over the OpenAI SDK, and the model name must not carry a provider prefix:

claimlens extract examples/extract/kepler22b.json \
  --extractor-base-api https://openrouter.ai/api/v1 \
  --extractor-model openai/gpt-5.6-luna

The same flag points at any OpenAI-compatible server, including a local one (--extractor-base-api http://localhost:11434/v1).

LiteLLM routing (optional extra, claimlens[litellm]) — omit the base URL and prefix the model with its provider instead. LiteLLM resolves the endpoint:

claimlens extract examples/extract/kepler22b.json --extractor-model openrouter/openai/gpt-5.6-luna

Prefer the direct endpoint for very new models: LiteLLM routing depends on the installed LiteLLM release knowing the model, while a base URL bypasses that lookup entirely. Either way the key comes from EXTRACTOR_API_KEY / CHECKER_API_KEY.

Documentation

• examples/README.md — the five-step walkthrough: what each command needs, what it produces, and a worked dataset with deliberate errors planted in it
• eval_data/msmarco/README.md — provenance of the human-labelled evaluation data, what was repaired in it, and what it can and cannot measure
• architecture.md — the full design document (layer model, data contract, error propagation)
• docs/request_strategies.md — how requests reach the model: capability discovery, retry behaviour, timeouts, and the settings you can tune
• docs/ragchecker.md — the RAG evaluation pipeline
• docs/faithfulness.md — ground-truth-free faithfulness checking

Scientific lineage

The methodology follows RefChecker (Amazon Science) and RAGChecker (claim-level RAG evaluation), with modernized components: explicit abstention handling, a per-item error taxonomy instead of silent nulls, claim deduplication, and variance reporting. It is an independent reimplementation, not a fork.

License

MIT