citewatch
Which websites do AI search answers cite, and how fresh are those pages?
citewatch asks AI models questions with web search on, records every source they cite, visits each cited page politely to find when it was last updated, and reports who gets cited, how fresh it is and how far to trust each number, with counts and 95% intervals throughout. This repository holds the tool and its first study: 100 pre-registered UK health questions asked of Claude Sonnet 5.5 and GPT-6.1 Sol, twice.
Dashboard · Post · Signal spec (PDF) · Data card · Pre-registration · Deviations · Decisions
It measures sourcing, not medical accuracy, and nothing here is medical advice.
What the first study found
| GPT-6.1 Sol | Claude Sonnet 5.5 | |
|---|---|---|
| Answers citing nhs.uk (run 1) | 83% (83/100; 95% CI 75%–90%) | 10% (10/100; 95% CI 5.0%–16%) |
| Citation weight held by its top 10 publishers (run 1) | 78% (n=100; 95% CI 75%–85%) | 24% (n=100; 95% CI 23%–31%) |
| Time-sensitive answers whose newest dated source is over a year old (both runs) | 43% (49/114; 95% CI 31%–55%) | 12% (14/114; 95% CI 5.8%–20%) |
- The two models rarely cite the same publishers for the same question: Jaccard similarity 10% (n=100; 95% CI 7.4%–13%) per question, run 1.
- The nhs.uk gap is separated in both runs after a Holm correction.
- How far to trust it: citewatch dated (or logged a reason for) 89% (745/841; 95% CI 86%–91%) of cited URLs, against a 90% target. On held-out pages it was within 7 days of the checked date on 67% (18/27; 95% CI 48%–81%) (agent-checked, owner-approved). In the support check, the two AI judges and the owner disagree, so the support rate is reported, not settled.
- These findings are directional: one country, two models, 100 questions.
Install
pip install citewatch # the core: report, score, dates, lookup
pip install 'citewatch[anthropic,openai]' # plus the provider SDKs for paid runs
uvx citewatch report --example # or try it without installing
v0.1.0 is not on PyPI yet; until it is, install from the repository with
pip install git+https://github.com/biplob819/ai-visibility. Python 3.11 or newer. See the quickstart (an example report in under
five minutes, no keys).
How it works
- Ask. Each frozen question goes to each model with its native web search, from one
location, under a cost cap; responses are stored verbatim (
citewatch ask). - Extract. Citations come only from the APIs' citation fields, with canonical URLs and a
publisher per URL (
citewatch extract). - Fetch and date. Each cited page is fetched once, obeying robots.txt at one request per
second per host; page bodies are held in memory and never stored. A date resolver reads
seven sources (visible text, schema.org, meta tags, microdata and RDFa, the search tool's
page age, sitemaps, HTTP headers) and never guesses (
citewatch fetch). - Score. Shares, freshness, staleness, overlap and stability with a question-cluster
bootstrap, Holm-corrected comparisons and Wilson intervals; domain cards for every cited
publisher, each with an improvement report (
citewatch score). - Report. The dashboard and post are built from
summary.json(citewatch report).
Two checks sit under the numbers: a support check (do cited pages state the claim? two AI
judges via EvalKit and the owner's labels) and a
held-out date check. Architecture: docs/assets/architecture.svg.
Rerun it
Look at the results with no keys and no spend:
git clone https://github.com/biplob819/ai-visibility && cd ai-visibility
uv sync
uv run citewatch report --example # the bundled study's dashboard
Run your own study (paid calls need the provider SDKs and your API keys):
uv sync --extra anthropic --extra openai
uv run citewatch init my-study --contact <a page site owners can reach you at>
# edit my-study/study.yaml and my-study/questions.jsonl, and set the API keys in my-study/.env
uv run citewatch doctor --study my-study # names any missing key or SDK
uv run citewatch ask --run 0 --dry-run --study my-study # cost estimate, no calls
# 1. A small paid smoke run measures what an answer really costs
uv run citewatch ask --run 0 --study my-study
uv run python scripts/measure_cost.py my-study --run 0 # writes my-study/cost_model.yaml
# 2. Before the study runs: freeze the questions and the metric definitions
# write my-study/preregistration.md, record the sha256 of it and of questions.jsonl in a
# decisions.md, and commit everything (a study run refuses a dirty or non-git folder)
# 3. The study runs, then the analysis
uv run citewatch ask --run 1 --study my-study # run 2 waits at least two hours
uv run citewatch extract --study my-study
uv run citewatch fetch --study my-study
uv run citewatch score --study my-study
uv run citewatch report --study my-study --title "my topic"
citewatch ask --run 1 --dry-run lists anything the run gate still needs before it will spend.
The gate and why each check exists are in the Tech Spec (§3 and gates G1, G2).
Cost. The flagship study's smoke run cost $0.79 and its two study runs (400 answers) cost
$15.03 in API calls; the dry run estimates yours before any call, and max_cost_usd in
study.yaml caps it. Fetching costs nothing but time. Thresholds in study.yaml can fail a CI
job when a metric crosses a bound (docs/thresholds.md).
Guides
| Guide | What it covers |
|---|---|
| Quickstart | the bundled study's dashboard in under five minutes, no keys |
| Run your own study | settings, the questions format, intents, country, models, the run gate |
| How dates are resolved | the seven date sources, the rules and their limits |
| Controlling cost | estimates, the hard cap, resume, what the flagship study cost |
| Thresholds in CI | rules that fail a CI job when a metric crosses a bound |
| Responsible use | politeness, no telemetry, privacy, not for gaming AI answers |
| Versioning | semver and the public API |
Responsible use
citewatch obeys robots.txt and a fixed politeness floor that no setting can loosen, sends
nothing anywhere except to the AI providers you configure and the cited sites, and is not a
tool for gaming AI answers (docs/responsible-use.md).
Repository map
| Path | What it holds |
|---|---|
src/citewatch/ |
the package: runners, extract, fetch, dates, metrics, support, cards, report, demo |
studies/uk-health/ |
the flagship study: questions, settings, frozen data, results |
docs/ |
pre-registration, deviations, decisions, signal spec, data-flow and architecture diagrams |
demo/ |
the Streamlit demo (lookup; live mode behind passkeys) |
specs/, docs/graph/ |
the build plan: one spec per node and the capability catalog |
Links
| Repository | https://github.com/biplob819/ai-visibility |
| Package | https://pypi.org/project/citewatch/ |
| Dashboard | https://biplob819.github.io/ai-visibility/ |
| Post | https://biplob819.github.io/ai-visibility/post/ |
| Signal spec | docs/signal-spec.pdf |
| Data card | DATA_CARD.md |
| Pre-registration | docs/preregistration.md |
| Deviations | docs/deviations.md |
| Decisions | docs/decisions.md |
| Changelog | changelog.md |
| Support judging | EvalKit |
scripts/check_links.py --online checks every link in the README and docs.
Licence
MIT (LICENSE). Contributions: CONTRIBUTING.md; security
reports: SECURITY.md.
Metadata
Release files for citewatch 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| citewatch-0.1.0.tar.gz | 241.0 kB | With provenance | Details for citewatch-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| citewatch-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for citewatch-0.1.0-py3-none-any.whl |
Total release size: 513.6 kB
Release files / citewatch-0.1.0.tar.gz
| Download URL | citewatch-0.1.0.tar.gz |
|---|---|
| Size | 241.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1c3ba54b966e57aade8d6d20d254e1ff5e74d30e46e6946c7154e18311a9c96e
|
|
BLAKE2b-256 checksum How to use checksums |
d9b3f6d2f2fd8425ee46dba15e8f2f9b1a50250124a592ce400f488e58ed70dc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logRelease files / citewatch-0.1.0-py3-none-any.whl
| Download URL | citewatch-0.1.0-py3-none-any.whl |
|---|---|
| Size | 272.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
64b3a1d58c17aa34278202e1bcd671c253bc060e8113e4b2858204df21b79ddc
|
|
BLAKE2b-256 checksum How to use checksums |
199d274dbe411dfb1728714f24a78a3285e1876fd0ba1ea69d71a3dbac48e649
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logSecurity for citewatch 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for citewatch-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.