refaudit

Verify the entries in a .bib against Crossref, arXiv and OpenAlex, and report the ones a human needs to look at.

Written for venues that run automated checks for hallucinated or malformed references. The failure it is built to catch is not a missing DOI — it is a DOI that resolves to a different paper than the entry claims. That one is invisible when you read the reference list, because the title and authors look fine and only the identifier is wrong.

Install

pip install refaudit

Python 3.10+. No runtime dependencies.

Latest development version:

pip install git+https://github.com/uw-share-lab/refaudit.git

Quick start

It reads the whole .bib and checks every entry in it — you point it at the file and it works through the lot, one entry at a time.

# every entry in the file
refaudit refs.bib --email you@uwaterloo.ca

# only the entries actually cited in the paper, which is usually what you want
refaudit refs.bib --email you@uwaterloo.ca --tex paper/sections --only-cited

--email is required. Crossref and OpenAlex give identified callers a separate, more reliable request pool, and it is the courtesy their documentation asks for. Set it once instead of typing it each time:

export REFAUDIT_EMAIL=you@uwaterloo.ca

Results are written to refaudit-out/ as reference_check.txt (readable) and reference_check.csv (sortable), and printed to stdout.

Working from Overleaf

Download the .bib (Menu → Download → Source, or just the file), then point --tex at the unzipped sections/ directory so --only-cited can tell which keys actually reach the PDF:

refaudit sample-base.bib --email you@uwaterloo.ca --tex sections/ --only-cited

What it reports

Exit status is 1 if there is at least one finding, 0 if not, 2 on a usage error — so it drops into CI or a pre-submission script.

Options

A run over a few hundred references takes minutes, because it is deliberately paced. Successful lookups are cached, so it is safe to interrupt with Ctrl-C and re-run — it picks up where it stopped.

The one design rule

"We could not check" and "this is wrong" are different answers and never collapse into each other.

This sounds obvious and is easy to get wrong. arXiv rate-limits whole networks; when that happens, a naive checker either silently passes the entry (false comfort, the worse failure) or falls back to a title search, finds something loosely related, and reports a mismatch (false alarm, which trains you to ignore it). Both are worse than saying "I could not check this one."

So every resolver returns exactly one of Found, NotFound, or Unavailable, and only Found can produce a negative verdict. UNVERIFIED results are listed separately from findings and are never cached, so a transient outage does not get baked into later runs.

Relatedly, evidence is weighted by strength: a title search returning something different is only a finding when there was no identifier to go on. Otherwise every arXiv-only workshop paper that Crossref does not index would be flagged.

The same care applies to DOIs, and for a while it did not. No registration agency speaks for the whole DOI system: Crossref registers most published literature, DataCite registers preprints and deposits — including every arXiv DOI (10.48550/*), Zenodo and figshare. Reading Crossref's 404 as "this DOI does not exist" reported 22 live preprints in a real bibliography as dead references. DEAD_DOI is now the one verdict confirmed against a second source: every agency has to disown the DOI and doi.org — which answers for all of them — has to report it unregistered. If the DOI resolves but nothing indexes it, that is UNVERIFIED; if doi.org cannot be reached, that is also UNVERIFIED. A reference is never called dead on one agency's silence.

Sources, and why there are several

No source is load-bearing. Every one of them can be missing, metered or unreachable without the run producing a false finding — that is the point of having several, not redundancy for its own sake.

Sources are tried in order of how much their answer is worth: identifier lookups before title searches, curated indexes before harvested ones. An identifier that resolves to a different paper ends the search — that disagreement is the finding. A weak title hit does not, because otherwise the first index to return anything at all would mask a better answer from the next.

If a source is down or metered, drop it and the rest still work:

refaudit refs.bib --email you@uwaterloo.ca \
  --resolvers crossref:doi,datacite:doi,doi:content,arxiv:id,dblp,crossref:title

Duplicates

The same work cited under two keys is invisible to the per-entry check: both copies resolve, both are correct, both report OK. Only comparing entries with each other finds it, so that runs as its own pass — offline, on entry order, and matching on DOI, arXiv ID (including the arXiv DOI form of the same ID) and near-identical titles. It costs nothing and works when every network source is refusing us.

Rate limiting

Each source declares the limit its own documentation specifies, next to the code that calls it:

Requests retry up to four times with exponential backoff and full jitter, so a transient failure does not become a false UNVERIFIED and simultaneous retries do not synchronise into a thundering herd. A definitive 4xx is never retried — it is an answer, not a failure.

Limits are held per host, not per resolver: two resolvers calling api.crossref.org are still one caller as far as Crossref is concerned, so they share a single bucket and the most cautious rate either declares. The same applies to the circuit breaker — when a host starts refusing us, every resolver that calls it backs off together. Raising --workers does not raise any of this: threads queue on the same buckets.

A 429 is treated as instruction rather than noise: Retry-After is honoured, the token bucket is permanently halved, and after repeated refusals a circuit breaker stops asking that host so the rest of the run still finishes. Retries use exponential backoff with full jitter.

--email is required because Crossref and OpenAlex give identified callers a separate, more reliable pool, and it is the courtesy their docs ask for.

Security

• HTTPS only. Plain-http URLs are refused, not silently upgraded.
• Bounded redirects, kept on https, so a redirect cannot downgrade transport or forward the mailto identifier somewhere unexpected.
• Response size cap and per-request timeouts on every call.
• XML is parsed with entity declarations refused, blocking billion-laughs and XXE. Uses defusedxml when installed, otherwise a hardened stdlib path; both raise the same exception type so callers cannot miss one.
• DOIs and arXiv ids are validated against a pattern before being interpolated into a request path, so a malformed field cannot steer the URL.
• No credentials in code. Optional API keys come from the environment and are sent as headers, never query parameters, so they stay out of logs.
• No runtime dependencies. This gets installed in a hurry near a deadline, often on a machine someone else administers; that is the wrong moment to widen the supply chain.

Troubleshooting

UNVERIFIED on an entry whose DOI you know is fine. The DOI resolves, but none of the indexes we can read hold metadata for it, so the reference is unchecked rather than wrong. This is the intended outcome, not a failure.

Lots of UNVERIFIED results. A source refused your network. arXiv in particular rate-limits by IP and will 429 an entire institution or VPN regardless of your own pace. These are not findings — the entries were simply not checked. Try again later, from a different network, or lean on OpenAlex, which indexes arXiv identifiers too:

refaudit refs.bib --email you@uwaterloo.ca --resolvers crossref:doi,openalex

A correct entry is flagged NOT_FOUND. Workshop papers, theses and tech reports are often in no citation index. If the entry has no DOI and no arXiv id there is nothing to verify it against; confirm it by hand and move on.

A correct entry is flagged TITLE_MISMATCH. This one is worth taking seriously: it means the DOI or arXiv id in your .bib resolves to a different paper. Usually the identifier was copied from the wrong row, or generated rather than looked up. Check the found line in the report against what you meant to cite.

Everything is SKIPPED. @misc and @online entries with no identifier cannot be checked. That is expected for datasets, blog posts and software.

API reference

Generated from the docstrings and published on every push to main: https://uw-share-lab.github.io/refaudit/refaudit.html

Build it locally with pdoc (included in the dev extra):

pdoc refaudit refaudit.checker refaudit.models refaudit.resolvers -o site --docformat google

Library use

from refaudit import Checker, default_resolvers, parse_file

entries = parse_file("refs.bib")
checker = Checker(default_resolvers("you@university.edu"))

for result in checker.check_all(entries):
    if result.verdict.is_finding:
        print(result.key, result.verdict.value, result.note)

Resolver is a Protocol: implement name, rate, can_handle and resolve to add a source, and pass it to Checker alongside the built-ins.

Development

pip install -e ".[dev]"
pytest          # offline: the suite uses fake resolvers and never hits a network
ruff check .
mypy src

main is protected: changes go through a pull request with a code-owner review, and CI must pass on Python 3.10-3.13. See CONTRIBUTING.md.

Repository settings that releases depend on are kept as code in .github/scripts/configure-repo.sh and are safe to re-run.

Licence

MIT.