citefact

Do your sources actually say what you claim?

Quickstart · How It Works · Usage · Limitations

_{A fabricated quote caught with the source's own words. Full report.}

LLMs made it trivial to produce manuscripts with fabricated citations, altered quotes, and claims the cited sources never made. Existing tools check that citations exist; citefact reads the full text of every cited source and checks that it supports what you wrote — then hands you a report with the evidence, before a reviewer finds it first.

🚀 Quickstart

# From a Zotero collection (Zotero 7+ running; no export needed):
uvx citefact check manuscript.md --zotero-collection "PhD/Chapter 3"

# Or from a BibTeX file and a folder of PDFs:
uvx citefact check manuscript.md --bib refs.bib --pdfs ./papers/

That's the whole setup: uvx fetches citefact, the report lands in ./citefact-report/report.html. The citation and quote checks are free and need no API key. The claims check uses an LLM with your own key — run uvx citefact setup once and it configures everything interactively, or pass --skip-claims to stay entirely keyless and offline.

🧭 Three Checks, Cheapest First

Every claim verdict (supported / partial / misrepresented / not_in_paper) carries a verbatim evidence quote, reasoning, and a confidence score. It is presented as an AI suggestion, never as ground truth: citefact reports, you decide.

✨ Why researchers keep it in the loop

• 📚 Zotero-native - point it at a collection and it reads metadata and attached PDFs through Zotero's local API; no exports, no Better BibTeX
• 🔒 Keyless mode - --skip-claims catches fabricated citations and altered quotes with zero network calls
• 🤖 Your LLM, your key - Anthropic, OpenAI, or fully local via Ollama, all through LiteLLM
• 📊 One-file report - self-contained HTML (no CDN, no requests), with every citation's status, word-level diffs for altered quotes, and one-click links to the manuscript and each source PDF
• ⚡ Cached everything - PDF conversions, claim extraction, and verdicts are content-addressed; editing one paragraph re-verifies only what changed, and an unchanged re-run costs $0 in seconds. Conversions go through pagebound, so a paper any other pagebound tool already converted is reused, never reconverted
• 🕵️ Private by construction - no telemetry; the only writes are the cache and the report, and the only network traffic is your LLM provider

⚙️ How It Works

flowchart LR
    A[manuscript.md] --> D[Citation parser]
    B[Zotero / refs.bib] --> E[Bibliography catalog]
    C[source PDFs] --> F[pagebound<br/>Docling, content-hash cache]
    D --> G[Citations check]
    E --> G
    F --> G
    G --> H[Quotes check]
    H --> I[Claims check<br/>LLM + caches]
    I --> J[report.html + report.json]

1. Parse - a deterministic parser extracts every in-text citation (narrative, parenthetical, multi-author lists, multi-citation groups) and resolves it against the bibliography by first-author surname + year.
2. Convert - source PDFs become text via pagebound, which runs Docling once per PDF and caches the result by content hash and converter version. Scanned PDFs are OCR'd, and the report says so.
3. Check - citations and quotes are verified deterministically; claims are extracted verbatim and judged per (claim, source) pair by the LLM, evidence included.
4. Report - a filterable single-file report.html plus a versioned report.json for CI.

🤖 For agents and LLM tools

An agent can drive citefact from --help alone: every command states what it writes, its exit codes and the shape of report.json, and citefact help COMMAND is the same text. The loop is three commands:

citefact doctor                                           # what is installed, exit 0 always
citefact check manuscript.md --bib refs.bib --pdfs papers/ --skip-claims --json --quiet > report.json
jq '.findings[] | select(.severity == "error") | {type, location, citation, quote}' report.json

Exit codes are the contract: 0 clean, 1 findings at or above --fail-on, 2 usage or execution error with the reason on stderr. Progress never touches stdout, so --json output is safe to pipe. Finding types are stable per level (orphan_citation, missing_source, quote_modified, claim_verdict, ...) and report.json carries schema_version; a claim_verdict with source_truncated: true was judged on a partial source and should be read as provisional; changes are listed in CHANGELOG.md. Converted source text is shared with pagebound, so an agent that wants to read a source itself can use pagebound path on the same cache.

📖 Usage

CI gating

citefact check manuscript.md --bib refs.bib --pdfs ./papers/ --skip-claims --fail-on error

Exit codes: 0 clean, 1 findings at or above --fail-on (default error), 2 execution error. Errors are the serious findings (fabricated citations, altered or unfound quotes, misrepresented/not_in_paper verdicts); --fail-on warning is stricter, --fail-on none just collects the report. Failed LLM calls surface as unverified findings and mark the run partial — never silently dropped.

Live progress

Long phases draw progress bars with ETA and running LLM cost on interactive terminals (plain lines in CI), and every run ends with a summary:

Verifying claims ━━━━━━━━━━━━━━╸━━  38/41  0:03:12  ETA 0:00:15  $0.31  smith2023

⚠️ Summary: 0 errors, 13 warnings (quotes 13) | cost $0.33 | 158.2s
Report: citefact-report/report.html

🤖 LLM Configuration

The easy path is one command:

uvx citefact setup

It asks for your provider, takes the API key with hidden input, validates it with a minimal call, and stores it in ~/.config/citefact/config.toml (owner-only permissions). Done — citefact check just works from then on.

Everything else is an override, in precedence order: --model / --provider flags > ANTHROPIC_API_KEY-style environment variables and CITEFACT_MODEL > the setup config file > the built-in default (anthropic/claude-sonnet-5). Any LiteLLM-supported provider works, including fully local:

uvx citefact check manuscript.md --bib refs.bib --model ollama/llama3.1

⚠️ Limitations

• Author-year (APA-like) citation styles only; numeric styles (Vancouver/IEEE) are on the roadmap
• Markdown manuscripts only in v0.1 (DOCX next; the manuscript itself cannot be a PDF)
• You supply the source PDFs; there is no automatic retrieval by DOI
• The first PDF conversion downloads the Docling environment (~1-2 GB, one-time) into the uv cache
• Claim verdicts are AI suggestions with the usual LLM failure modes; that is why every verdict carries its evidence

Roadmap: DOCX manuscripts → LaTeX (\cite{}) → numeric citation styles → stable JSON schema and JOSS submission. report.json already carries schema_version and is the contract for integrations; schema changes are listed in CHANGELOG.md.

git clone https://github.com/hearthresearch/citefact
cd citefact
uv sync
uv run pytest              # unit tests, LLM mocked
uv run pytest --run-evals  # opt-in LLM eval fixtures (costs money)

📄 License

MIT

🤝 Acknowledgments

• LiteLLM for provider-agnostic LLM access
• pagebound and Docling for PDF-to-Markdown conversion with page provenance
• ZotSeek - related tool: AI-powered semantic search for Zotero

citefact: Do your sources actually say what you claim? Built by José Fernandes