refaudit

Verify the entries in a .bib against Crossref, arXiv and OpenAlex, and report the ones a human needs to look at.

Written for venues that run automated checks for hallucinated or malformed references. The failure it is built to catch is not a missing DOI — it is a DOI that resolves to a different paper than the entry claims. That one is invisible when you read the reference list, because the title and authors look fine and only the identifier is wrong.

Install

pip install refaudit

Python 3.10+. No runtime dependencies.

Latest development version:

pip install git+https://github.com/uw-share-lab/refaudit.git

Quick start

It reads the whole .bib and checks every entry in it — you point it at the file and it works through the lot, one entry at a time.

# every entry in the file
refaudit refs.bib --email you@uwaterloo.ca

# only the entries actually cited in the paper, which is usually what you want
refaudit refs.bib --email you@uwaterloo.ca --tex paper/sections --only-cited

--email is required. Crossref and OpenAlex give identified callers a separate, more reliable request pool, and it is the courtesy their documentation asks for. Set it once instead of typing it each time:

export REFAUDIT_EMAIL=you@uwaterloo.ca

Results are written to refaudit-out/ as reference_check.txt (readable) and reference_check.csv (sortable), and printed to stdout.

Working from Overleaf

Download the .bib (Menu → Download → Source, or just the file), then point --tex at the unzipped sections/ directory so --only-cited can tell which keys actually reach the PDF:

refaudit sample-base.bib --email you@uwaterloo.ca --tex sections/ --only-cited

What it reports

An arXiv identifier counts wherever the entry keeps it: eprint, or free text such as journal={arXiv preprint arXiv:2506.08872}, which is what Google Scholar exports look like. | UNVERIFIED | not checked — a source was unreachable, or the DOI is real but unindexed. Says nothing about the entry | | SKIPPED | @misc/@online/@phdthesis and similar with no identifier; nothing to check against | | OK | resolved and consistent |

Exit status is 1 if there is at least one finding, 0 if not, 2 on a usage error — so it drops into CI or a pre-submission script.

Options

A run over a few hundred references takes minutes, because it is deliberately paced. Successful lookups are cached, so it is safe to interrupt with Ctrl-C and re-run — it picks up where it stopped.

The one design rule

"We could not check" and "this is wrong" are different answers and never collapse into each other.

This sounds obvious and is easy to get wrong. arXiv rate-limits whole networks; when that happens, a naive checker either silently passes the entry (false comfort, the worse failure) or falls back to a title search, finds something loosely related, and reports a mismatch (false alarm, which trains you to ignore it). Both are worse than saying "I could not check this one."

So every resolver returns exactly one of Found, NotFound, or Unavailable, and only Found can produce a negative verdict. UNVERIFIED results are listed separately from findings and are never cached, so a transient outage does not get baked into later runs.

Relatedly, evidence is weighted by strength: a title search returning something different is only a finding when there was no identifier to go on. Otherwise every arXiv-only workshop paper that Crossref does not index would be flagged.

The same care applies to DOIs, and for a while it did not. No registration agency speaks for the whole DOI system: Crossref registers most published literature, DataCite registers preprints and deposits — including every arXiv DOI (10.48550/*), Zenodo and figshare. Reading Crossref's 404 as "this DOI does not exist" reported 22 live preprints in a real bibliography as dead references. DEAD_DOI is now the one verdict confirmed against a second source: every agency has to disown the DOI and doi.org — which answers for all of them — has to report it unregistered. If the DOI resolves but nothing indexes it, that is UNVERIFIED; if doi.org cannot be reached, that is also UNVERIFIED. A reference is never called dead on one agency's silence.

Sources, and why there are several

No source is load-bearing. Every one of them can be missing, metered or unreachable without the run producing a false finding — that is the point of having several, not redundancy for its own sake.

Sources are tried in order of how much their answer is worth: identifier lookups before title searches, curated indexes before harvested ones. An identifier that resolves to a different paper ends the search — that disagreement is the finding. A weak title hit does not, because otherwise the first index to return anything at all would mask a better answer from the next.

If a source is down or metered, drop it and the rest still work:

refaudit refs.bib --email you@uwaterloo.ca \
  --resolvers crossref:doi,datacite:doi,doi:content,arxiv:id,dblp,crossref:title

Duplicates

The same work cited under two keys is invisible to the per-entry check: both copies resolve, both are correct, both report OK. Only comparing entries with each other finds it, so that runs as its own pass — offline, on entry order, and matching on DOI, arXiv ID (including the arXiv DOI form of the same ID) and near-identical titles. It costs nothing and works when every network source is refusing us.

Rate limiting

Each source declares the limit its own documentation specifies, next to the code that calls it:

Requests retry up to four times with exponential backoff and full jitter, so a transient failure does not become a false UNVERIFIED and simultaneous retries do not synchronise into a thundering herd. A definitive 4xx is never retried — it is an answer, not a failure. Every request sent is paced, retries and redirect hops included: a redirect is a real request to a real server, so it costs the same as any other.

Limits are held per host, not per resolver: two resolvers calling api.crossref.org are still one caller as far as Crossref is concerned, so they share a single bucket and the most cautious rate either declares. The same applies to the circuit breaker — when a host starts refusing us, every resolver that calls it backs off together. Raising --workers does not raise any of this: threads queue on the same buckets.

A 429 is treated as instruction rather than noise: Retry-After is honoured, the token bucket is halved, and after repeated refusals a circuit breaker stops asking that host so the rest of the run still finishes. Retries use exponential backoff with full jitter.

If a 429 carries a Retry-After longer than a minute, that is an answer rather than a delay: the request fails immediately and the host is stood down for the period it asked for, so the rest of the run skips it instead of rediscovering the same refusal entry by entry. Anything that source would have answered comes back UNVERIFIED — no verdict, never a finding.

The halving is a penalty the run can work off. Each success afterwards edges the rate back up by a twentieth of the host's ceiling, and a penalty never takes it below a sixteenth of that ceiling. Decrease is multiplicative and recovery is additive — quick to yield, slow to re-probe — so a burst of refusals early on costs you seconds rather than leaving the remaining few hundred entries crawling for the rest of the run. Rates a service publishes are different: those are a ceiling, and recovery never climbs past one.

--email is required because Crossref and OpenAlex give identified callers a separate, more reliable pool, and it is the courtesy their docs ask for.

Several people running it at once

Everyone runs under their own --email, so each of you is a separate identified caller with your own allowance, your own backoff and your own circuit breaker. One person hitting a 429 does not slow anybody else down, and no amount of parallel use by a lab gets an individual blocked. There is nothing to configure.

Several runs by one person at once

Two terminals, a shell loop or a cluster job array are a different matter: those are one caller as far as Crossref is concerned, and a per-process limit would send it a multiple of the rate we promised.

So the token bucket for each host is shared by every refaudit you are running, through a small state file under your own cache directory (~/.cache/refaudit/, or %LOCALAPPDATA% on Windows). Four runs at once share one allowance rather than taking four. A 429 in any of them slows all of them, and the recovery afterwards is shared too.

It is per user, not per machine, which is deliberate: two people on a shared server are still two identified callers and should not be throttling each other.

Everything about it degrades. If the state file cannot be created, read, written or locked, each run paces itself exactly as it did before — correct on its own, uncoordinated with the others — so the worst case is the behaviour that came first. --no-shared-pacing, or REFAUDIT_NO_SHARED_PACING=1, turns it off entirely and keeps refaudit from writing anything outside --out.

Two runs in the same directory also share the cache file, which is safe. Writes are atomic, each flush merges what is already on disk rather than overwriting it, and the read-merge-write happens under a lock, so neither run loses the other's entries. Where flock is not honoured, as on some network mounts, the lock falls back to an atomic lock directory with two rules that keep it from being worse than no lock at all: one left behind by a process that died is taken over after 30 seconds, and waiting for it is bounded, after which the run carries on unlocked rather than not at all. If you would rather keep the runs entirely separate, give each one its own --out.

Security

• HTTPS only. Plain-http URLs are refused, not silently upgraded.
• Bounded redirects, kept on https, so a redirect cannot downgrade transport or forward the mailto identifier somewhere unexpected.
• Response size cap and per-request timeouts on every call.
• XML is parsed with entity declarations refused, blocking billion-laughs and XXE. Uses defusedxml when installed, otherwise a hardened stdlib path; both raise the same exception type so callers cannot miss one.
• DOIs and arXiv ids are validated against a pattern before being interpolated into a request path, so a malformed field cannot steer the URL.
• No credentials in code. Optional API keys come from the environment and are sent as headers, never query parameters, so they stay out of logs.
• No runtime dependencies. This gets installed in a hurry near a deadline, often on a machine someone else administers; that is the wrong moment to widen the supply chain.

Troubleshooting

UNVERIFIED on an entry whose DOI you know is fine. The DOI resolves, but none of the indexes we can read hold metadata for it, so the reference is unchecked rather than wrong. This is the intended outcome, not a failure.

Lots of UNVERIFIED results. A source refused your network. arXiv in particular rate-limits by IP and will 429 an entire institution or VPN regardless of your own pace. These are not findings — the entries were simply not checked. Try again later, from a different network, or lean on OpenAlex, which indexes arXiv identifiers too:

refaudit refs.bib --email you@uwaterloo.ca --resolvers crossref:doi,openalex

A correct entry is flagged NOT_FOUND. Workshop papers, theses and tech reports are often in no citation index. If the entry has no DOI and no arXiv id there is nothing to verify it against; confirm it by hand and move on.

A correct entry is flagged TITLE_MISMATCH. This one is worth taking seriously: it means the DOI or arXiv id in your .bib resolves to a different paper. Usually the identifier was copied from the wrong row, or generated rather than looked up. Check the found line in the report against what you meant to cite.

You want to see what it is actually doing. Add -v. Every request, retry, rate-limit change and circuit-breaker trip goes to stderr, which is the fastest way to tell a slow network apart from a service that is refusing you:

refaudit refs.bib --email you@uwaterloo.ca -v

Diagnostics go to stderr and the report to stdout, so you can keep them apart:

refaudit refs.bib --email you@uwaterloo.ca -v 2> refaudit.log

Everything is SKIPPED. @misc and @online entries with no identifier cannot be checked. That is expected for datasets, blog posts and software.

API reference

Generated from the docstrings and published on every push to main: https://uw-share-lab.github.io/refaudit/refaudit.html

Build it locally with pdoc (included in the dev extra):

pdoc refaudit refaudit.checker refaudit.models refaudit.resolvers -o site --docformat google

Library use

from refaudit import Checker, default_resolvers, parse_file

entries = parse_file("refs.bib")
checker = Checker(default_resolvers("you@university.edu"))

for result in checker.check_all(entries):
    if result.verdict.is_finding:
        print(result.key, result.verdict.value, result.note)

Resolver is a Protocol: implement name, rate, can_handle and resolve to add a source, and pass it to Checker alongside the built-ins.

Importing refaudit configures no logging and prints nothing — it attaches only a NullHandler, so level and destination stay your application's decision. To see what the network is doing, handle the refaudit logger yourself:

import logging
logging.basicConfig(level=logging.INFO)
logging.getLogger("refaudit").setLevel(logging.DEBUG)

WARNING carries the things worth acting on — rate-limit penalties, retries and circuit-breaker trips. DEBUG adds one line per request.

Development

pip install -e ".[dev]"
pytest          # offline: the suite uses fake resolvers and never hits a network
ruff check .
mypy src

The suite is hermetic so CI cannot be failed by a busy upstream. That has a cost worth being honest about: a green run proves the code behaves correctly against fixtures, and says nothing about whether it still parses what Crossref actually sends. One test does check that, and it is opt-in:

REFAUDIT_LIVE_EMAIL=you@uni.edu pytest tests/test_live.py -v -m live

It checks seven references against the real services: one correct DOI, one arXiv id, one DataCite DOI, a real DOI deliberately paired with the wrong title, a DOI that does not exist, and an entry with no identifier at all. What it asserts is the thing that must never break, which is that a correct reference is never reported as a finding. A source being unreachable is allowed to turn any of them UNVERIFIED, because that is a fact about the network rather than the entry.

It also runs weekly from .github/workflows/live.yml, away from pull requests, so an upstream having a bad day costs a notification rather than a blocked merge. Worth running by hand before cutting a release: 0.4.1 exists because it caught something 127 hermetic tests could not.

main is protected: changes go through a pull request with one approving review, and CI must pass on Python 3.10-3.13. See CONTRIBUTING.md.

Repository settings that releases depend on are kept as code in .github/scripts/configure-repo.sh and are safe to re-run.

Licence

MIT.