upheld

Checks whether an AI answer's citations actually support its claims.

AI search tools and deep-research agents now mostly cite real, working links, but the cited page often doesn't say what the answer claims: one 2026 study of 14 models found only 39–77% of cited facts were actually in the source. Link checkers can't see it. upheld splits an answer into claims, finds the relevant passage in each cited source, and says which claims the sources really back.

How it's meant to be used

One engine, several ways in. They all call the same run_audit().

What it reports

Each citation gets one verdict, and each claim gets the best verdict among its citations:

A page upheld can't read live, because it's blocked or dead, is read from the Wayback Machine's newest copy instead and judged like a live page. The report says so and gives the copy's date.

Every report names its verifier and threshold, because measured citation accuracy swings widely with the verifier (3% to 18% on identical outputs in one study).

Quickstart

pip install "upheld[models]"

import upheld

report = upheld.audit("The Eiffel Tower is 330 metres tall [1]. It was designed by Frank Lloyd Wright [1].", ["https://en.wikipedia.org/wiki/Eiffel_Tower"])
for claim in report.claims:
    print(claim.verdict, claim.text)

supported The Eiffel Tower is 330 metres tall.
unsupported It was designed by Frank Lloyd Wright.

upheld.audit takes an answer with [1]-style markers or inline links, and its sources in order: URLs to fetch, or the text itself. It returns an AuditReport with a verdict, a support score and the evidence for every claim. To audit a provider's response as it comes back from the API, see Use it in your code.

The first run downloads about 4 GB of models from Hugging Face: the two that the default ensemble verifier combines (lytang/MiniCheck-Flan-T5-Large, KRLabsOrg/lettucedect-base-modernbert-en-v1) and the retrieval embedder (BAAI/bge-small-en-v1.5). verifier="minicheck" uses one model. Plain pip install upheld leaves out torch and the models, and runs only verifier="lexical", retrieval="bm25", which is much weaker. The most accurate verifier, ensemble+judge, also needs pip install "upheld[models,judge]", which builds llama.cpp from source, and downloads the 8.7 GB Granite Guardian GGUF on first use. upheld serve needs the api extra (FastAPI and uvicorn): pip install "upheld[models,api]".

The minimum versions are as old as upheld works with (pydantic 2.0, httpx 0.21; torch 2.6 and transformers 4.48 for the models), so it installs into an existing project without forcing upgrades.

The same from the command line:

upheld audit answer.md                       # markdown answer with a reference list; fetches the URLs
upheld audit response.json --from openai     # a provider's API response, saved as JSON
upheld audit answer.md --json                # full report
upheld serve                                 # POST /audit on http://127.0.0.1:8000 (needs upheld[api])

From a clone

The examples, evals and tests are in the repository:

git clone https://github.com/warun7/upheld && cd upheld
uv sync --extra models                             # base deps + MiniCheck + LettuceDetect + embedding model (torch)
uv run upheld audit examples/glp1.json             # inline source text
uv run upheld audit examples/eiffel.md             # markdown answer with a reference list; fetches URLs
uv run upheld eval evals/smoke.jsonl               # verifier vs labels, with a threshold sweep
(cd web && npm ci && npm run build)                # optional: then `upheld serve` also hosts the site at /
uv run pytest

uv sync --all-extras adds the judge. RELEASING.md covers publishing to PyPI.

Input

• JSON (AuditRequest): {"question": "...", "answer": "...", "sources": [{"id": "1", "url": "...", "text": "..."}]}. A source with text is used as-is; otherwise its url is fetched. Optional cited_spans, [{"start": 0, "end": 40, "cited": ["1"]}], mark character ranges of the answer as cited; each is checked as one claim, in place of the sentences it overlaps.
• Markdown/text: an answer with [1]-style markers or inline [text](url) links, optionally ending in a reference list ([1] Title - https://...). Use - to read from stdin.
• A provider's response: see below.

Use it in your code

Hand upheld the response your code already gets from OpenAI, Anthropic, Perplexity or Gemini with web search on. It reads the provider's own citation data, so there's nothing to reformat:

import upheld
from openai import OpenAI

question = "How tall is the Eiffel Tower?"
response = OpenAI().responses.create(model="gpt-6-astra", tools=[{"type": "web_search"}], input=question)
report = upheld.audit(response, question=question)
print(f"{report.summary.supported_claims} of {report.summary.cited_claims} cited claims supported")

The other providers' responses go in the same way:

import anthropic
from google import genai
from perplexity import Perplexity

# Anthropic: Messages with the web search tool
message = anthropic.Anthropic().messages.create(
    model="claude-sonnet-5-5",
    max_tokens=2048,
    messages=[{"role": "user", "content": question}],
    tools=[{"type": "web_search_20250305", "name": "web_search"}],
)
report = upheld.audit(message, question=question)

# Gemini: Google Search grounding, from the Interactions API or generate_content
interaction = genai.Client().interactions.create(model="gemini-3.8-flash", input=question, tools=[{"type": "google_search"}])
report = upheld.audit(interaction, question=question)

# Perplexity: the Agent API, or Sonar
response = Perplexity().responses.create(effort="low", input=question)
report = upheld.audit(response, question=question)

A response can be the SDK's object, its JSON as a dict, or a JSON string. upheld tells the provider from its shape; provider="gemini" names it. upheld.from_openai(response), and likewise from_anthropic, from_perplexity and from_gemini, return the AuditRequest upheld builds, so you can see the claims and sources before auditing.

A cited range too short to be a claim on its own, such as a name, is widened to the sentence it's in. Sentences outside every cited range are still reported, as uncited when they read as claims. Claim offsets refer to the answer text upheld assembled, upheld.from_…(response).answer.

Gemini's redirect links stop working after a while, so audit a Gemini response soon after you get it. An expired link is reported unverifiable, since the page it pointed to is unknown.

Load the models once

upheld.audit keeps the models from its last call loaded. To hold them yourself, or to audit from async code, make an Auditor:

auditor = upheld.Auditor()  # verifier="ensemble", retrieval="auto": a few seconds and about 4 GB to load

for answer, docs in batch:
    report = auditor.audit(answer, docs)  # docs: your retrieved texts, cited as [1], [2], ...

report = await auditor.audit_async(response)  # in an event loop, such as a web server

run_audit(request, verifier, embedder=...) is still there for full control.

From the command line and over HTTP

upheld audit response.json --from openai     # or anthropic, perplexity, gemini; auto, the default, tells them apart
pip install "upheld[api]"                    # once, for upheld serve
upheld serve &
curl -s localhost:8000/audit/auto -H 'content-type: application/json' -d @response.json

POST /audit/{provider} takes the response as its body, and the question as an optional ?question= parameter, and returns the same report as POST /audit.

How it works

1. Parse: split the answer into sentences, map [1], [1, 2], [^1] and inline links to sources, and pull out a trailing reference list. Lead-ins and questions aren't treated as claims. When a provider's response marks exactly which text a source backs (Anthropic's cited blocks, Gemini's grounding ranges), each of those ranges is a claim instead.

2. Fetch: download cited URLs (5 MB cap) and extract the main text: trafilatura for web pages, pypdf for PDFs, with a PDF's wrapped lines and words hyphenated across lines joined back up. A 404/410 or unresolvable domain is checked against the Wayback Machine: archived means link rot, never archived means likely fabricated. 401/403/429/5xx and timeouts only mean "couldn't read", and so does a page with under 300 characters of text, which is a cookie wall, bot check or JavaScript shell rather than the article, or a scanned PDF with no text layer. For a dead page or one that couldn't be read, upheld then reads the newest copy the Wayback Machine archived with HTTP 200 (one lookup and one fetch), extracts it the same way, with the same 300-character floor, and judges claims against it like a live page. The source is reported as archived (or link_rot if it's dead), with the copy's date and address, and each citation of it says it was checked against the copy. Only a working copy counts as archived, so a URL the Wayback Machine has seen only as a 404 is likely fabricated. A scanned PDF, a PDF over the cap or an unsupported file type isn't looked up, since its copy is the same file.

3. Retrieve evidence: chunk each source into ~120-word windows and pick the top 3 per claim, or the top 10 for the judge ensemble, which turns more of the page into fewer false alarms (see Fresh 2026 benchmark). Hybrid mode (the default when the dense extra is installed) ranks passages with both BM25 and bge-small-en-v1.5 embeddings and merges the rankings with reciprocal rank fusion, so paraphrases aren't missed. --retrieval bm25 uses keywords only.

4. Verify: the default verifier combines three signals with a logistic regression whose four numbers were fit on 2,000 AttributionBench training rows:

   • MiniCheck-Flan-T5-Large (770M), trained to judge whether a document supports every part of a claim. Long sources are split into ~500-word chunks and the claim takes its best chunk.
   • Word overlap, with every number in the claim required to appear in the evidence.
   • LettuceDetect, run once per source over the whole answer (it was trained on full responses; scored as isolated sentences it flagged true claims with 0.89 hallucination probability). Its flagged tokens, or else the mismatched numbers, become the highlighted spans.

   --verifier ensemble+judge adds a fourth signal, refit with the other three: Granite Guardian 3.3 8B, an LLM trained to judge whether a response is grounded in its documents, run locally with llama.cpp. It reads the claim against each ~1,500-word chunk of the evidence and takes the best.

5. Entity check: flag named entities in the claim that appear nowhere in the source. This gets its own verdict, wrong_entity, because it's a heuristic with a known false-alarm rate (see How good is it).

6. Combine: a claim citing several sources that individually fail is re-checked against them together (multi-hop claims).

Fetching is guarded against SSRF (no private, loopback or metadata addresses, checked on every redirect) since a hosted API fetches URLs taken from untrusted text. --allow-private lifts this for local use.

How good is it

Three kinds of test: a public benchmark of human-labeled citations, edited claims aimed at each failure upheld exists to catch, and a fresh benchmark of 2026 answers. The first two give the verifier the evidence directly, so parsing, fetching and retrieval aren't exercised. The fresh benchmark runs the whole pipeline on real pages.

AttributionBench

AttributionBench (Li et al., 2024): 3,296 test claims, each with the evidence it cited and a human label for whether that evidence fully supports it, from seven datasets. Stanford-GenSearch and AttrScore-GenSearch are real answers from Bing Chat, Perplexity, YouChat and NeevaAI. Classes are balanced, and the metric is macro-F1 averaged over subsets, as in the paper. Every upheld threshold below was picked on the dev split, never on test.

ID = in-distribution subsets (ExpertQA, Stanford-GenSearch, AttributedQA, LFQA); OOD = BEGIN, AttrScore-GenSearch, HAGRID. Every subset is in evals/results/attributionbench.json.

What this says:

• The default verifier went from 64.4 to 74.4 ID macro-F1, above every zero-shot system in the paper and 4 points below the best fine-tuned one. On OOD it's within about a point of the best fine-tuned model. It runs locally in roughly 0.3–1 s per claim on an Apple M5 Pro.
• The ensemble is lightly supervised, not zero-shot. Its three weights and bias were fit on 2,000 rows of AttributionBench's train split: different rows from the test sets, but the same datasets. The clean zero-shot comparison is MiniCheck alone at its default threshold of 0.5: 73.3 ID and 79.6 OOD, level with GPT-4 on ID from a 770M model.
• Most of the ensemble's gain over MiniCheck is on OOD (+3.8), where answers mostly copy wording from their evidence and word overlap is a strong signal. On ID it adds 0.5, which is within noise.
• An LLM judge closes more than half of the remaining ID gap. Adding Granite Guardian 3.3 8B (IBM, Apache 2.0) as a fourth signal lifts ID from 74.4 to 76.7, 1.8 below the best fine-tuned model. Most of the gain is on real search-engine answers: Stanford-GenSearch goes from 71.8 to 78.6. OOD dips 0.8, all of it on BEGIN, where answers copy their evidence and word overlap already did the work. The judge costs an 8.7 GB model and, since its ensemble reads 10 passages per claim, about 2.5–3 s per claim in all, so it's opt-in: --verifier ensemble+judge.
• The judge alone is about as good as MiniCheck on ID and better on search-engine answers : 74.2 ID at the threshold picked on dev, and 75.6 at its default of 0.5, which involves no fitting at all. It reads the claim without the user's question: shown the question, it also judged whether the claim answered it, and its AUC on 300 dev rows fell from 0.85 to 0.76.
• ExpertQA is still the hard part. Expert-written claims against long technical evidence: 59.3 with the judge, level with GPT-4's 59.2. Every model in the paper scores below 62.
• The entity check costs about 2 points here. It turns 6% of the supported test claims into false alarms (names the source writes differently: abbreviations, partial names, translations), and AttributionBench has few wrong-entity errors to make up for it. The edited claims below show what it buys.

Edited claims

evals/perturbations.py takes AttributionBench claims that humans labeled as supported, edits each one so its evidence no longer supports it, and checks whether upheld notices. 150 edits per type. Catch rates count only claims whose unedited version passes, so a verifier that flags everything earns nothing.

What this says:

• Wrong numbers are solved. The ensemble catches every one, because the word-overlap signal requires each number in the claim to appear in the evidence.
• A name the source never mentions is caught 87% of the time by the ensemble alone and 98% with the entity check. That's why the check stays on despite its benchmark cost, but reports its own verdict, wrong_entity: treat it as a prompt to look, not proof.
• Pinning a fact on the wrong party is the weak spot. When the swapped-in name also appears in the evidence (a claim credits a result to another drug, person or study the same page mentions), the default ensemble catches only 35%. MiniCheck alone catches 57%; the ensemble loses ground because word overlap finds every word of the edited claim in the source. The judge nearly doubles the ensemble's rate, to 64%, but this is still the case upheld handles worst.
• Combining sources works. Two supported claims joined into one sentence and cited [1][2] aren't fully supported by either source. Re-checking against both together marks 83% of them supported; without that step all of them would be flagged. No single source ever passed one on its own, except with word overlap (6%).

The edits are synthetic and the claims come from AttributionBench, so these numbers say whether each check does its job, not how often each failure happens in real answers.

Fresh 2026 benchmark

25 current questions, answered on 31 October and 1 November 2026 by Claude, OpenAI and Gemini agents with their own web search (how, under Evals). upheld split each answer into claims and found 641 pairs of a claim and a page it cites. Each pair is labeled supported, partial, unsupported or can't judge: 639 by Claude Sonnet 5.5 subagents following labeling-guide.md, 2 by hand. The labelers saw the question, the claim, the cited URL and upheld's retrieved passages, but not which agent wrote the answer or upheld's verdict, and read the full page whenever the passages didn't settle it. GPT and Gemini relabeled a sample of 150 to check Claude's labels (below).

Perplexity has one answer (4 citations) so far and isn't ranked. The full table is evals/fresh/scoreboard.md. Its dead-link counts come from a re-audit on 3 November and differ in two places: Gemini shows 7, all one FDA label PDF that answers upheld with a 404 on most tries but serves the file to a browser, and Claude shows 5, because the Wayback Machine check failed for one of its dead links that day.

upheld against these labels: re-audited on 3 November, and again on 4 November for the 40 answers citing a page it couldn't read, on the 554 citations whose page it could read, the default pipeline agreed with the label 75% of the time. It flagged 90% of the citations labeled partial or unsupported, and 70% of its flags were right. 74 of those citations were read from a Wayback Machine copy because the page blocked upheld or was dead; it agreed with the label on 50 of them. The 4 November re-audit fetched those answers' other pages again too, and 10 citations upheld had read the day before were blocked this time with no readable copy (PubMed, the BMJ, FindLaw, FourWeekMBA), while 1 went the other way. With only the archived copies' verdicts added to the 3 November audit, it agrees on 75% of 563 citations, with precision 0.70 and recall 0.90.

With the judge: each verifier on the 564 labeled citations whose page upheld can read, with the page fetched and the passages retrieved the way upheld does it (evals/fresh/verifiers.py, without the entity check, pages fetched on 3 November). 62 of them, citing 25 pages that block upheld, time out or are block pages, were read from a Wayback Machine copy on 4 November:

The judge ensemble halves the false alarms on good citations and catches a few fewer bad ones. Part of that comes from reading more of the page:

On the 62 citations read from an archived copy, the judge ensemble scores 82.3 macro-F1 and flags 7 of 32 good citations, close to how it does overall. The default ensemble does worse there, 66.9 and 16 of 32: most of those pages are long statistical releases and trial reports full of figures. The misses don't come from stale copies: the copies taken before 2026 are of journal papers and news stories that don't change, and pages that do (BLS's current CPI release, Perplexity's help center) were copied in October or November 2026.

With more passages the default ensemble flags fewer good citations, but it misses bad ones at the same pace (91% caught at 3 passages, 82% at 10), so it keeps 3. The judge ensemble at 10 still catches 86% and flags 18% of good citations against 30% at 3, so it takes 10. The judge alone does best at 3. The settings were compared on these labels, so 83.7 is a little optimistic; all three are shown. The judge's lead holds when GPT or Gemini does the labeling instead of Claude (below), but all three labelers are LLMs, so some of the gain may still be LLMs agreeing with each other rather than the judge being right.

What this says:

• About half of each agent's citations fully back their sentence. Most of the rest are partial: the page has part of the sentence but not a figure, a second fact or a conclusion drawn from it. Citations that back nothing at all are 7–17%.
• No clear winner. Each agent's share has a margin of roughly ±6–9 points, so OpenAI and Claude are level. Gemini's lower share is almost all partial: it is rarely flat wrong, but its sentences often say more than the cited page does.
• Dead links are rare: 7 of 641 citations, 6 of them Claude's.
• Pages upheld can't read were the biggest gap, and archived copies close most of it. 142 citations (22%) cited a page that blocked upheld, timed out, served a block page or was dead. Reading the Wayback Machine's newest copy instead leaves 50 (8%) without a verdict: 45 pages were read from copies taken between July 2024 and 2 November 2026, 19 of them in October or November 2026. What remains is mostly pages that block automated fetches and were never archived, or whose archived copy is a block page too.
• upheld works as a first filter on real answers. The default ensemble misses few bad citations, but about 3 in 10 of its flags are false alarms. The judge ensemble halves the false alarms on good citations, from 38% to 18%, and 83% of its flags are right. Two causes are fixed: block pages (cookie walls, VPN and bot checks, JavaScript shells) were judged as if they were the page, which made 7 good citations false alarms, and the supporting sentence often wasn't among the 3 passages retrieved. Of the judge ensemble's 51 remaining false alarms, 46 had the supporting sentence among their 10 passages, so what's left is mostly the verifier: a number restated in other units (4,310 mg of sodium as 10.8 g of salt), a sentence that leans on the one before it ("It also held in June and July."), or several facts in one sentence.

Checking Claude's labels

GPT (gpt-6-luna through the Codex CLI) and Gemini (Gemini 4 Argon through the Antigravity CLI) relabeled 150 citations, sampled in proportion to each agent and each of Claude's labels, under the same guide and the same blinding, and without seeing Claude's labels. Both got the passages plus the cited page's text, fetched once so that both read the same document. Full results: evals/fresh/crosscheck.md.

• Claude agrees with each of them as often as they agree with each other. Where GPT and Gemini give the same verdict, Claude matches on 118 of 128 citations, and couldn't read the page for 6 more. The other 4 are citations Claude passed and both others called partial, all claims that say a little more than the page: "the DOJ argued" where the page says the court found, or "2.4 to 2.5 ppm" where the page gives 2.4.
• No sign that Claude favors its own answers. Claude's citations get 47% supported from Claude, 45% from GPT and 47% from Gemini. GPT is the strictest labeler on each ranked agent (40% supported overall, against 45% from Claude and from Gemini), and Gemini's citations rank last under all three labelers.
• upheld's numbers hold. On the 129 sampled citations whose page upheld can read, the judge ensemble scores 84.5, 83.4 and 85.9 macro-F1 against Claude's, GPT's and Gemini's labels, and the default ensemble scores 77.1, 78.7 and 75.1.

Caveats: the labels come from LLMs, not people. The cross-check rules out a bias peculiar to Claude, not one that all three models share, and only 2 citations have hand labels (Claude matched one; on the other it couldn't read the paywalled page). 57 citations (9%) are can't judge, mostly pages that block automated fetches; some blocked pages were read through an archived copy or the paper's abstract and others weren't. And these are agents with search following the same instructions, not the consumer search products.

Evals

uv run python evals/attributionbench.py score minicheck --device mps   # cache one score per example, per split
uv run python evals/attributionbench.py report                         # dev-split thresholds, the ensemble, the tables above
uv run python evals/perturbations.py ensemble                          # the edited-claim tests

score runs lexical, lettuce, minicheck or judge over AttributionBench's train sample, dev and both test splits and caches the scores in evals/results/scores/, so report can compare and combine verifiers without rerunning them. report also prints the weights that EnsembleVerifier hard-codes for both ensembles, with and without the judge.

Fresh 2026 benchmark (evals/fresh/, results under How good is it). AttributionBench's search-engine answers are from 2023. This collects answers to 25 current questions, converts them to [n] markers, and labels each citation blind, with the provider and upheld's verdict hidden.

Each answer comes from a vendor's own agent using that vendor's own web search, with the same instructions: answer as a search assistant would, cite with [n] markers, use nothing but search.

• Claude: Claude Sonnet 5.5, as Claude Code subagents with the WebSearch tool.
• OpenAI: gpt-6-luna through the Codex CLI with OpenAI's web search. It runs in an empty read-only sandbox, without the user's MCP servers.
• Gemini: Gemini 3.8 Flash through the Antigravity CLI (agy) with its search_web tool. Google's redirect links are resolved to the real URLs when the answer is collected.

The OpenAI and Gemini answers run on a ChatGPT or Google subscription, not a paid API. The collector rejects any answer whose agent used a tool other than search, or cited a source it didn't list. Answers that come back in their own citation format (a provider's API, say) don't measure quite the same thing, so these rows compare agents with search rather than the consumer search products. Perplexity has no subscription CLI. It goes through OpenRouter at about a cent per question, and only its first answer is collected so far.

uv run python evals/fresh/collect.py openai gemini --limit 25   # needs codex and agy, logged in
uv run python evals/fresh/label.py prepare      # audit every answer, fetching the cited pages; resumable
uv run python evals/fresh/label.py label        # label by hand, s/p/u/x per citation; resumable
uv run python evals/fresh/label.py scoreboard   # per-provider table + upheld's agreement with the labels
uv run python evals/fresh/verifiers.py pages     # fetch each labeled citation's page once, as upheld does
uv run python evals/fresh/verifiers.py score judge && uv run python evals/fresh/verifiers.py report   # each verifier vs the labels, at 3, 6 and 10 passages
uv run python evals/fresh/crosscheck.py sample && uv run python evals/fresh/crosscheck.py pages    # 150 citations, their pages fetched once
uv run python evals/fresh/crosscheck.py label gpt && uv run python evals/fresh/crosscheck.py label gemini && uv run python evals/fresh/crosscheck.py report

The labels in labels-claude.jsonl came from Claude subagents given labeling-guide.md and one blinded batch of items each; each "supported" or "partial" label carries a quote from the page. Hand labels in labels.jsonl override them. The output, scoreboard.md, answers both "how often do each tool's citations hold up?" and "how often does upheld agree with the labels?", on 2026 answers with real fetching and retrieval. items.jsonl, the pairs with their retrieved passages, holds text copied from the cited pages, so it isn't committed; prepare rebuilds it.

verifiers.py scores each verifier against the labels without the rest of the pipeline. It fetches each labeled citation's page once, the way upheld does, retrieves the top 3, 6 and 10 passages per claim, and caches every member's score for each setting in scores/, so the ensembles and settings can be compared without rerunning the models. The fetched pages, in sources/, aren't committed.

crosscheck.py checks the Claude labels. GPT and Gemini relabel a stratified sample of 150 citations through their subscription CLIs, with no tools at all. Antigravity can't open a URL, so each cited page is fetched once and both labelers get the same text in the prompt. A page that can't be read is taken from the Wayback Machine, then from the copy Claude read; text under 300 characters counts as a block page. A batch is rejected and retried if its labeler used a tool or skipped an item. The labels are in labels-gpt.jsonl and labels-gemini.jsonl, and the report is crosscheck.md. The fetched pages, in pages/, aren't committed either.

upheld eval scores any verifier against a JSONL of hand labels (claim, evidence, label, and ideally the full answer), with a threshold sweep. evals/smoke.jsonl is 8 rows that check the command works; it is not a benchmark.

evals/retrieval_check.py checks retrieval: for 10 paraphrased claims against the live Eiffel Tower Wikipedia article, it reports where the supporting passage ranks. BM25 puts it in the top 3 for 9/10 claims, hybrid for 10/10. The claims are hand-written, so this is a regression check, not a benchmark.

Known limitations

• One-step inferences fail without the judge. In examples/glp1.json the source reports "the mean change in body weight ... was -14.9%" and the answer says "an average weight loss of 14.9%". The default ensemble scores that 0.42 and marks it unsupported, as is the matching placebo claim. --verifier ensemble+judge passes both (0.70 and 0.71). examples/eiffel.md still marks the true claim "intended to stand for only 20 years" unsupported with the judge (0.44): retrieval finds "slated for dismantling in 1909", but nothing connects that to 20 years.
• Wrong-party claims slip through about two times in three when the wrong name is also in the source, and about one time in three with the judge (see Edited claims).
• Contradictions in the source's own words can pass without the judge. "The Great Wall of China is visible from the Moon with the naked eye", cited to its Wikipedia article, scores 0.57 and is marked supported, though the retrieved passage says it cannot be seen: word overlap and LettuceDetect outvote MiniCheck. The judge ensemble marks it unsupported (0.45).
• Numbers are matched as digits, not quantities. "It stands 410 metres tall" passes against the Eiffel Tower article (0.64) because the page gives the base as 410 ft. It passes with the judge too (0.52).
• ExpertQA-style claims (long, technical, expert-written) score 57.8 macro-F1, or 59.3 with the judge, so expect more mistakes on specialist content.
• Entity check is a heuristic (capitalized phrases + word frequency), not NER. It false-alarms on names the source writes differently and misses lowercase entities.
• Speed. The ensemble takes roughly 0.3–1 s per claim per source on an Apple M5 Pro and is several times slower on CPU. MiniCheck dominates. The judge ensemble reads 10 passages per claim, so it takes about 2.5–3 s per claim per source with Metal (the judge about 2 s of that, against 0.75 s on 3 passages), and needs about 9 GB of free memory. --verifier lexical is instant but much weaker.
• No OCR. A scanned PDF with no text layer comes back unverifiable, and so does a PDF over the 5 MB cap.
• Only short block pages are caught. A page with under 300 characters of readable text is reported as unreadable rather than judged: in the fresh benchmark those were all cookie walls, VPN or bot checks, or JavaScript apps whose text loads in the browser. A longer stand-in, such as a paywall teaser with the article's first paragraph, is still read as the page.
• Sentence-level claims. A sentence with two facts is judged as one unit.
• Some servers answer 404 to anything that isn't a browser, so upheld can call a live page a dead link. In the fresh benchmark an FDA label PDF did this on most tries, which made 7 of Gemini's citations look dead. They're judged against the PDF's archived copy now, but still counted as dead links.
• Archived copies can be out of date. A page upheld can't read live is judged against the Wayback Machine's newest copy, which can be older than the page the answer cited. The report gives the copy's date. In the fresh benchmark the copies were taken between July 2024 and 2 November 2026; the old ones were of journal papers and news stories that don't change.
• A blocked page with no readable archived copy stays unverifiable. In the fresh benchmark that's 50 of 641 citations (8%), on 25 pages: 14 never archived, 8 whose copy is a block page too or a PDF over the cap.
• Some sites reject generic user agents. The default names this repo; set UPHELD_USER_AGENT to your own project URL if you deploy upheld (Wikipedia returns 403 instead of 404 for missing pages without one).

Roadmap

Phase 1: auditor

• Hybrid retrieval (BM25 + a small embedding model) done
• Benchmark the verifiers on AttributionBench done
• Stronger verifier, threshold calibrated on a held-out split done: MiniCheck + ensemble, 64.4 → 74.4 ID macro-F1
• Edited-claim tests for each failure type done
• Run the fresh 2026 benchmark done: 641 citations from Claude, OpenAI and Gemini agents, 41–51% fully supported, and upheld flags 90% of the rest. GPT and Gemini relabeled a 150-citation sample and agree with Claude on 92–93% of verdicts, with no sign of self-preference. Next: Perplexity (about $0.25 of OpenRouter credit), and hand labels on the citations the labelers disagree on
• Cut false alarms on real pages done: block pages are reported unreadable, and the judge ensemble reads 10 passages per claim, flagging 18% of good citations against the default's 38%
• LLM judge for hard claims done: Granite Guardian 3.3 8B as an opt-in fourth signal, 74.4 → 76.7 ID macro-F1, wrong-party swaps caught 35% → 64%, fresh-label agreement 76% → 84% with 10 passages per claim. Next: fine-tuning MiniCheck on AttributionBench's train split (the paper's best fine-tuned model reaches 78.5 ID), and a faster judge so it can be the default
• Speed: MiniCheck in bf16, and batching claims across sources
• PDF sources done: read with pypdf, which made 9 of the 10 PDFs cited in the fresh benchmark readable. Next: OCR for scans
• Read archived copies of pages upheld can't read done: a page that blocks upheld, times out, serves a block page or is dead is read from the Wayback Machine's newest copy, which cut the fresh benchmark's unverifiable citations from 142 (22%) to 50 (8%)
• Installable package and provider responses done: pip install upheld, upheld.audit(), and OpenAI, Anthropic, Perplexity and Gemini responses read with their own citation data. Next: an MCP server so coding agents can check the citations in their drafts, then a pytest helper and a GitHub Action
• Web app done: landing page and playground in web/, with the site live at upheld.varunsg.com. Next: deploy the playground with the hosted API, then a browser extension

Phase 2: train a search agent to cite honestly

• Qwen3-4B + LoRA, GRPO, BM25 over Wikipedia, NQ + HotpotQA
• Baseline outcome-only reward → measure citation faithfulness with this auditor
• Add a faithfulness reward from this verifier; watch for reward hacking (copying, refusals); fix with sentence-level credit or grounding-gated accuracy