truthcheck
Open-world fact verification for AI claims, the web-search complement to halluguard.
Status: v0.1, working. Pipeline ships: Exa search backend, NLI verifier (lexical fallback when sentence-transformers not installed), SQLite cache, atomic claim splitter. Sibling to
adaptmem+halluguard+claimcheck.
The problem this solves
halluguard answers: "Is this claim supported by the documents I gave you?"
That's enough when you control the corpus (your shop's catalog, your company's internal docs, your codebase). It is not enough when:
- An LLM cites a figure ("Türkiye nüfusu 85 milyon").
- An LLM dates an event ("Bitcoin halving was in May 2024").
- An LLM names a person ("Alice Novak is the lead developer of Project X").
- An LLM repeats a recent news fact ("OpenAI released o4-mini in March 2026").
Halluguard can't answer because the ground truth lives on the open web,
not in the user's corpus. That's truthcheck's job.
Design constraints
- Stay composable. Truthcheck is a sibling, not a replacement.
halluguard.Guard.check(answer)→ corpus-grounded verdicttruthcheck.WebFactChecker.check(claim)→ open-web verdict- Caller decides which to invoke (or both, in series).
- Never silently dilute halluguard's positioning. Halluguard says "no LLM, no internet, deterministic." Truthcheck explicitly says "yes LLM (probably), yes internet, probabilistic." Honest naming.
- Backend-agnostic. Brave Search, Exa, Bing, DuckDuckGo, your internal corporate Confluence + Notion, anything that returns ranked snippets should plug in.
- Cost-aware. Web search APIs cost money. Truthcheck must
- tell the caller a USD estimate per claim before issuing requests
- cache aggressively (claim text → result, TTL configurable)
- support
dry_run=Trueto preview without API spend.
Sketch of the API
from truthcheck import WebFactChecker
checker = WebFactChecker(
backend="exa", # default; "brave" also supported
api_key=os.environ["EXA_API_KEY"],
trusted_domains=["wikipedia.org", "*.gov", "*.edu"],
cache_dir="~/.cache/truthcheck",
)
verdict = checker.check(
claim="Türkiye nüfusu 85 milyon",
n_sources=5,
)
# Verdict {
# status: SUPPORTED | UNSUPPORTED | CONTRADICTED | INCONCLUSIVE,
# confidence: 0.0, 1.0,
# sources: [
# Source(url="https://www.worldometers.info/...", snippet="...", score=0.91),
# Source(url="https://en.wikipedia.org/wiki/Demographics_of_Turkey", ...),
# ...
# ],
# atomic_claims: ["country: Türkiye", "metric: population", "value: 85 million"],
# cost_usd: 0.0007,
# cache_hit: False,
# }
v0.1 decisions (closed)
- Default backend: Exa (Brave's free tier was removed)
- Splitter: regex-based, deterministic, spacy/LLM in v0.2
- Verifier: NLI cross-encoder; lexical fallback when sentence-transformers absent
- Cache: SQLite under
~/.cache/truthcheck - Contradiction: INCONCLUSIVE + all sources surfaced
- Recency:
as_oftimestamp stamped on every verdict
Open for v0.2
- Turkish / multilingual NLI model
- spacy or small LLM for compound claim splitting
- DDG / SearXNG backend (no API key)
- Redis cache backend
Composition with the cluster
answer + corpus → halluguard.Guard.check()
│
▼
SUPPORTED? yes ─→ trust=high, done
│
no (claim isn't in corpus)
│
▼
answer claims → truthcheck.WebFactChecker.check()
│
▼
open-web verdict
Bigger picture: cluster gives the consumer a "belge → halluguard, dünya
→ truthcheck" pipeline so closed-world and open-world claims can both be
verified through one call site (a future helper in claimcheck).
What this repo is NOT
- Not a replacement for halluguard. Halluguard handles the case where you have a corpus. Don't use truthcheck where halluguard fits.
- Not a search engine. It's a verification layer that uses search engines as a substrate. Bring your own backend.
- Not a fact-database. It doesn't ship knowledge graphs. Every verdict is computed at request time against live sources.
- Not a guarantee. Open-world fact-checking is an active research area; FEVER state-of-the-art is around 75% F1. Truthcheck reports confidence, never asserts truth.
License
MIT
Install
pip install "truthcheck[brave]" # Brave backend
pip install "truthcheck[nli]" # NLI verifier (sentence-transformers)
Set EXA_API_KEY or BRAVE_API_KEY env var before use.
This is a draft. Atakan to review, sharpen the open questions, and decide whether to push public + commit to the v0.1 milestone.
Metadata
Release files for nakata-truthcheck 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| nakata_truthcheck-0.1.1.tar.gz | 16.4 kB | Without provenance | Details for nakata_truthcheck-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| nakata_truthcheck-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for nakata_truthcheck-0.1.1-py3-none-any.whl |
Total release size: 33.0 kB
Release files / nakata_truthcheck-0.1.1.tar.gz
| Download URL | nakata_truthcheck-0.1.1.tar.gz |
|---|---|
| Size | 16.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8602ea66b516d31588f9ff0f6df4fbf6090ddf6e36bd43d78706919f595fa4ef
|
|
BLAKE2b-256 checksum How to use checksums |
905175f0b63a1cedf298000f0d636afcf8f15f74cfddb99b980d82b714864b49
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.13
|
Release files / nakata_truthcheck-0.1.1-py3-none-any.whl
| Download URL | nakata_truthcheck-0.1.1-py3-none-any.whl |
|---|---|
| Size | 16.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ccf0b600b5a2bd358ea45e04aa96fc4b25ea875b982982f45875ccb63a4a1c43
|
|
BLAKE2b-256 checksum How to use checksums |
643a91b6eaaee23321b557166b9feadabd238410fff2135744960773826cde8b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.13
|
Security for nakata-truthcheck 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.