citefinder

OpenAlex (default) + Crossref reference lookups with local JSONL caching.

A small Python library + CLI for verifying academic references against the OpenAlex and Crossref APIs. Every lookup is appended to an append-only JSONL log so repeated queries (across verification passes or sessions) are served from the cache. Negative results (404s) are cached too, so known-missing DOIs aren't re-hit.

OpenAlex is the default source: it merges Crossref + Unpaywall + ORCID + ROR

• repository sources, so it covers what Crossref alone is missing — arXiv DOIs (10.48550/arXiv.*), other preprints, repository deposits — and frequently has richer metadata (abstracts, full author lists, affiliations) for records that exist in both. Crossref is still available via the crossref subcommand for its own workflows (book-chapter lookup, the canonical published-deposit metadata).

Configuration

OpenAlex works without authentication, but a free API key gives you higher limits and tier-specific endpoints. Both Crossref and OpenAlex honor a mailto for their polite pools (faster responses, higher quotas).

• OpenAlex docs: https://developers.openalex.org/
• Sign up / generate an OpenAlex key: https://openalex.org/login?redirect=/settings/api-key

The CLI reads its settings from two TOML files plus the environment:

• Project config — citefinder.toml, or a [tool.citefinder] table in pyproject.toml, found by walking up from the working directory. The nearest one wins; in one directory citefinder.toml beats pyproject.toml. Meant to be committed: where a repo's caches go, which polite-pool address it uses.
• User config — ~/.config/citefinder/config.toml (honors $XDG_CONFIG_HOME). Per machine: credentials and personal defaults.

Both take the same keys; every one is optional:

cache_dir = "data/citefinder"   # relative: to this file's directory

[openalex]
api_key = "your-openalex-key"   # user config or .env only — see below
mailto = "you@example.com"
max_retries = 3                 # optional — see "Rate limits and retries"
min_interval = 0.1

[crossref]
mailto = "you@example.com"
max_retries = 3
min_interval = 0.34             # optional — the polite-pool rate is the default

[openlibrary]                   # books by ISBN — see "Book metadata by ISBN"
mailto = "you@example.com"      # sent in the User-Agent, as Open Library asks
max_retries = 3
min_interval = 1.0

[loc]                           # Library of Congress; no contact parameter
max_retries = 3
min_interval = 1.0

In pyproject.toml the same keys sit under [tool.citefinder], [tool.citefinder.openalex], [tool.citefinder.crossref], and so on.

Precedence, highest first:

cache_dir is the one directory every cache path derives from. doi, search, and the crossref subcommands write <cache_dir>/<source>.jsonl; verify writes its output under <cache_dir>/<bib-dir>[-<bib-stem>]/<source>/ (<bib-dir> is the directory holding the .bib; see CLI usage). When nothing sets it, lookups use ~/.cache/citefinder/ and verify uses data/citefinder/ under the working directory. A relative cache_dir in a config file resolves against that file's directory, so data/citefinder in a repo's citefinder.toml means the repo's data/citefinder from any working directory inside it; a relative --cache-dir or CITEFINDER_CACHE_DIR resolves against the working directory, as flags do. An explicit --cache (lookups) or --out (verify) always wins.

Keep secrets out of the project config. It is meant to be committed, so an api_key there is ignored with a warning — put it in .env or the user config. mailto is fine to commit. The user config is plain text; if your environment is shared, chmod 600 ~/.config/citefinder/config.toml so it's only readable by you.

citefinder config prints each resolved setting with its source (flag, env, project, user, or default) and the paths the lookups and verify would write to — the place to look when a cache landed somewhere unexpected.

Library users: pass cache_path=..., api_key=..., and mailto=... to the client constructors explicitly; resolve_cache_path(source, cache_dir) returns the same <cache_dir>/<source>.jsonl the CLI uses. Config files are CLI-only (they shouldn't be a surprise side effect of importing the library).

The API key is sent as Authorization: Bearer ..., never as a URL parameter, so it doesn't land in cache keys, logs, or referer headers.

Install

uv add citefinder

Or for development:

git clone https://github.com/gitronald/citefinder
cd citefinder
uv sync

Library usage

OpenAlex (default)

from citefinder import (
    OpenAlexClient,
    datacite_registrar,
    is_arxiv_doi,
    reconstruct_abstract,
)

openalex = OpenAlexClient(
    cache_path="~/.cache/citefinder/openalex.jsonl",
    mailto="you@example.com",  # opts into OpenAlex's polite pool — faster, higher quota
)

# Single DOI (works for arXiv DOIs that Crossref doesn't index)
work = openalex.lookup_doi("10.48550/arXiv.2410.21554")

# Title-only search — tuned for citation verification. Handles OpenAlex's
# curly-apostrophe quirk and strips filter-reserved punctuation that would
# 400 the request, so straight ASCII inputs match curly-quoted indexed titles.
hits = openalex.search_title("Backstabber's Knife Collection", rows=3)

# Free-text search across titles + abstracts (noisier; prefer search_title
# for citation lookup)
hits = openalex.search("fact-checking large language models", rows=3)

# OpenAlex stores abstracts as an inverted index — reconstruct to plain text
abstract = reconstruct_abstract(work) if work else None

# Helpers for routing logic: DataCite DOIs are never in Crossref
assert is_arxiv_doi("10.48550/arXiv.2410.21554")
assert datacite_registrar("10.5281/zenodo.123") == "Zenodo"

The mailto argument is optional but recommended: it puts requests into OpenAlex's polite pool for faster responses. The cache key strips mailto so changing it doesn't invalidate prior entries.

Crossref

from citefinder import CrossrefClient

client = CrossrefClient(
    cache_path="~/.cache/citefinder/crossref.jsonl",
    mailto="you@example.com",  # opts into Crossref's polite pool — faster, higher quota
)

# Single DOI
work = client.lookup_doi("10.1126/science.aap9559")
print(work["title"][0])

# Bibliographic search (author + title + year)
hits = client.search_bibliographic("Wolfowicz hate speech meta-analysis", rows=3)

# Book chapter via {book_doi}.{NNN} pattern
chapter = client.lookup_book_chapter("10.1017/9781108890960", 5)

Crossref and OpenAlex both honor mailto for their polite pools; the cache key strips it on either side, so rotating the email doesn't invalidate prior entries.

OpenAlex's schema differs from Crossref. The record shapes are declared as TypedDicts in citefinder.models (CrossrefWork, OpenAlexWork, and their nested parts; the model is rough and grows as fields get used), and citefinder drift <cache.jsonl> lists what the records in a cache carry that the model does not. Quick map of the fields verify uses:

The two sources regularly disagree on a work's year because they index different events: Crossref's published-print tracks the printed issue/volume year, while OpenAlex's publication_year often collapses to the online-first date — or, for books, to a precursor work (e.g., the dissertation a monograph grew out of; a type of dissertation or posted-content next to a journal/monograph DOI is the giveaway). Treat a year mismatch as a flag for review, and default to the final printed record: the journal volume year, or the publisher's first-published edition year for books.

Rate limits and retries

Both clients retry a request that comes back 429 (rate limited) or 502/503/504 (gateway errors), up to max_retries times (default 3), before raising the original HTTPError. The wait honors the response's Retry-After header when present — both the delta-seconds and the HTTP-date form — and otherwise backs off exponentially from backoff_base (default 1 s: 1 s, 2 s, 4 s, each plus up to half a step of jitter). Any single wait is capped at max_wait (default 60 s). Other 4xx responses raise immediately, and 404 is still cached as None.

Requests can also be paced: min_interval is the minimum number of seconds between the start of consecutive requests from one client instance. Each client defaults to the rate its API advertises. OpenAlexClient uses 0.1 (10/s), well inside OpenAlex's 100/s ceiling. CrossrefClient resolves its own: 1.0 (1/s) anonymously, or 0.34 (3/s) when contact information puts the request in Crossref's polite pool — a mailto argument, or a mailto: in the User-Agent, both count. Supplying a mailto therefore speeds the client up rather than leaving it to find the higher rate by hitting 429s.

Passing min_interval explicitly — including 0, unpaced — overrides the resolved default. Cache hits are not requests and are never paced. All four knobs must be finite and non-negative; anything else raises ValueError at construction.

Where those numbers come from, and how to re-measure them: docs/crossref.md for Crossref's advertised rates and the polite pool, docs/openalex.md for OpenAlex's daily credit budget, what each lookup costs, and why pacing is not its binding constraint.

openalex = OpenAlexClient(
    cache_path="~/.cache/citefinder/openalex.jsonl",
    max_retries=5,  # extra attempts after the first; 0 disables retrying
    backoff_base=1.0,  # first backoff step when there is no Retry-After
    max_wait=60.0,  # ceiling on any single wait
    min_interval=0.2,  # seconds between requests
)
openalex.lookup_doi("10.48550/arXiv.2410.21554")
print(openalex.retries)  # retries so far on this instance
print(openalex.network_calls)  # requests that reached the network

Each retry logs one warning on the citefinder logger naming the status, the attempt, and the wait. The retries counter on the client tallies them for the run, and citefinder verify prints it in its summary line.

Alongside it, network_calls counts the requests that missed the cache and went out, one per call regardless of how many times it was retried. It is the counter to read for cache effectiveness: the cache's size only moves when a new key is stored, so a refetch — or a client with no cache at all — leaves it flat while requests are still going out.

Error responses are never cached: nothing is written until a 2xx or 404 arrives, so a run that hit the rate limit can simply be re-run once the limit clears. There is no cache line to purge.

On the CLI, --max-retries and --min-interval are accepted by doi, search, verify, and the crossref subcommands. OPENALEX_MAX_RETRIES / OPENALEX_MIN_INTERVAL and CROSSREF_MAX_RETRIES / CROSSREF_MIN_INTERVAL are the environment fallbacks, and max_retries / min_interval under [openalex] / [crossref] in config.toml the lowest-priority ones.

Bib verification

A .bib file can be parsed and verified against either source end-to-end:

from citefinder import (
    OpenAlexClient,
    Source,
    parse_entries,
    verify_entry,
)

source = Source(name="openalex", client=OpenAlexClient(cache_path="cache.jsonl"))

for entry in parse_entries(open("refs.bib").read()):
    result = verify_entry(entry, source)
    print(result.key, result.status, result.matched_doi)

Each Result reports a Status (matched / probable / mismatch / doi-not-found / not-indexed / unmatched / skip-source / error) plus the four signals — title, year, first-author surname, container — that drove the verdict. BibCitation and Work are the canonical shapes; crossref_to_work and openalex_to_work adapt source-specific JSON into Work. See citefinder/signals.py for the signal-check thresholds.

Book metadata by ISBN

Most trade books have no DOI, so Crossref and OpenAlex return unmatched or a same-word article, and the fields a book citation needs — full title with subtitle, publisher or imprint, place of publication, year — go unchecked. book_record runs the chain a librarian would: the ISBN to an Open Library edition, its Library of Congress Control Number to the Library of Congress catalog record (the Cataloging in Publication data printed on the copyright page), merged field by field with the catalog winning:

from citefinder import LocClient, OpenLibraryClient, book_record

record = book_record(
    "978-0-374-61932-9",
    OpenLibraryClient(cache_path="openlibrary.jsonl", mailto="you@example.com"),
    LocClient(cache_path="loc.jsonl"),
)
record.full_title  # 'Enshittification: why everything suddenly got worse and ...'
record.place  # BookField(value='New York', source='loc')
record.publisher.value  # 'MCD, Farrar, Straus and Giroux' — the imprint and its parent

Every field is a BookField carrying its source: loc for the catalog record, openlibrary for an edition a library catalog fed, or openlibrary:retailer for one fed only by retailer feeds (Amazon, Better World Books) — usually right, never confirmed, so it is reported as a lead rather than a value to apply. A source that lacks a field leaves it None; silence never counts as agreement. When Open Library has no edition or no LCCN, the catalog is asked by ISBN directly. LocClient talks to the catalog's SRU gateway (plain HTTP; it speaks no TLS) and caches the MARCXML parsed into JSON, so both caches stay JSONL; parse_marcxml and marc_book are exposed for reading other MARC records.

verify_entry(entry, source, books=BookLookup(openlibrary, loc)) uses the chain for @book, @inbook, and @incollection entries that carry an isbn and no doi (method="isbn"), checking title, year, and first contributor as the DOI path does and filling Result.suggestions with what the bib should add or change (a missing location, a dropped subtitle) and where each value came from. A bib publisher that names only the imprint is not flagged. A chapter entry is checked as its container: booktitle and editor stand in for title and author. Entries without an isbn keep their source verdict and get Open Library search hits as candidates, each with an isbn to confirm.

Bib ↔ table

A .bib file can be loaded into a wide polars DataFrame (one row per entry, one column per field) for inspection or bulk editing, then serialized back:

from citefinder import bib_to_table, table_to_bib

df = bib_to_table(open("refs.bib").read())  # key, entry_type, then fields alphabetical
new_bib = table_to_bib(df)  # back to .bib, null cells skipped

bib_to_table lowercases field keys (DOI → doi) and stores the entry kind in entry_type to avoid collision with the literal type field that some entries carry (e.g., SSRN papers set type = {SSRN Scholarly Paper}). table_to_bib requires key and entry_type columns and serializes the rest in column order. The round-trip is lossless on field values and entry types; the original within-entry field order and any source-file @string/@comment blocks are not preserved. Both refuse input they cannot round-trip: bib_to_table raises ValueError on a bib field literally named key or entry_type rather than overwriting it, and table_to_bib raises on a value with unbalanced braces, since the .bib it would write drops every later field on re-parse.

CLI usage

# OpenAlex (default)
citefinder doi 10.48550/arXiv.2410.21554 --mailto you@example.com
citefinder search "Backstabber's Knife Collection" --rows 3

# Crossref
citefinder crossref doi 10.1126/science.aap9559 --mailto you@example.com
citefinder crossref search "Wolfowicz hate speech meta-analysis" --rows 3
citefinder crossref chapter 10.1017/9781108890960 5

# .bib verification
citefinder verify refs.bib                               # full pipeline (defaults to OpenAlex)
citefinder verify refs.bib --source crossref             # ...or against Crossref
citefinder verify refs.bib --out path/to/output/dir/     # custom output directory
citefinder verify refs.bib --no-fallback                 # skip the shared cache; read only this run's
citefinder verify refs.bib --books                       # + check @book/@inbook/@incollection by ISBN

# Books by ISBN: Open Library, then the Library of Congress
citefinder isbn 978-0-374-61932-9                        # merged record, each field with its source
citefinder isbn 9780374619329 --no-loc                   # stop at Open Library

# .bib ↔ table
citefinder bib-to-table refs.bib                            # wide polars table to terminal
citefinder bib-to-table refs.bib --csv > refs.csv           # ...or CSV to stdout
citefinder bib-to-table refs.bib --fields title,year,doi    # subset of columns
citefinder table-to-bib refs.csv                            # CSV back to .bib on stdout
citefinder table-to-bib refs.csv --out refs.regen.bib       # ...or to a file

# Rate limits (see "Rate limits and retries" above)
citefinder ratelimit                                        # quota the last request saw; asks only if none
citefinder ratelimit --refresh                              # take a reading now (zero credits on OpenAlex)
citefinder ratelimit --source crossref                      # ...for Crossref instead

# Configuration (see "Configuration" above)
citefinder config                                           # resolved settings, their sources, and cache paths
citefinder drift data/citefinder/paper/openalex/openalex.jsonl  # keys the cached records carry that citefinder.models lacks

# Cache maintenance (see "Cache maintenance" below)
citefinder cache stats                                      # what every cache under cache_dir holds
citefinder cache merge                                      # dry run: consolidate into <cache-dir>/<source>.jsonl
citefinder cache merge --write                              # ...and apply it
citefinder cache compact data/citefinder/openalex.jsonl     # dedupe one file to a line per key

# Claude Code skill (see "Claude Code skill" below)
citefinder skill                                            # print the skill body
citefinder doc --list                                       # reference docs the body loads
citefinder install                                          # stub into ~/.claude/
citefinder install --local                                  # ...or the current repo
citefinder install --check                                  # status table; exits 1 unless ok

verify walks each entry: if a doi field is present it resolves the DOI; otherwise it searches by author + title + year. Each result is checked against four signals (title, year, first-author surname, container) and bucketed by status. Output goes to <cache_dir>/<bib-dir>[-<bib-stem>]/<source>/ — data/citefinder/ under the working directory when no cache_dir is set (see Configuration): a <source>.jsonl cache and a structured results.json. <bib-dir> is the name of the directory holding the .bib, and the -<bib-stem> suffix is added for any file not named refs.bib, so paper/refs.bib lands in paper/ and paper/extra.bib in paper-extra/ — same-named bibliographies in different directories never share an output directory. Re-running is cheap — every cache hit is served from disk.

Read results.json by method × status:

• method=doi with mismatch — a real defect: the bib's own DOI resolves to a different work.
• method=doi with probable — the DOI resolved but the title disagrees, or too few fields could be checked; usually a deficient bib title or missing fields, occasionally a typoed DOI that lands on a related paper.
• method=doi with matched and a non-empty note — the DOI resolved and the other signals confirm the work, but one field disagrees. With OpenAlex that is usually the source's metadata (a title truncated at the colon, a series name instead of the booktitle, a preprint year), not a reason to rewrite the entry.
• method=search with matched and a non-empty matched_doi — a DOI candidate for an entry that lacked one.
• method=search with mismatch or probable — usually a wrong-work false positive (books, reports, and other sources the index carries poorly), not a reason to rewrite the entry.
• unmatched (or skip-source for @online/@misc) with a "title too short" note — a bib title of fewer than three words cannot select a search hit; pick from candidates by hand.
• not-indexed — a Crossref run skips DataCite DOIs (arXiv, Zenodo) without a request, since Crossref never indexes them; verify those with --source openalex.
• unmatched, skip-source, and doi-not-found — noise unless they cluster around one publisher or entry type.

With --books, a book entry that has an isbn and no doi is checked against its catalog record instead of searched for (method=isbn), and each such result carries suggestions — fields the bib should add or change, each with the source it came from and whether it is confirmed by a catalog or only offered by a retailer-fed record. The Open Library and Library of Congress caches land beside the source cache in the output directory. isbn prints the same merged record for one ISBN, with the URL of each source's record. See Book metadata by ISBN.

bib-to-table and table-to-bib are inverses: the first turns a .bib into a wide table (terminal view by default, --csv for piping), the second reads such a CSV back into a .bib. Useful for spreadsheet-style review or bulk edits before regenerating the file. The round-trip is lossless on data; within-entry field order and source-file formatting are not preserved. Input that cannot round-trip — a bib field named key or entry_type, a cell with unbalanced braces — is refused with an error and exit 1 rather than written out with data silently lost.

CLI arguments

• --cache PATH — JSONL cache path. Defaults to <cache_dir>/openalex.jsonl for top-level commands and <cache_dir>/crossref.jsonl for crossref subcommands, with cache_dir falling back to ~/.cache/citefinder/. Separate files so sources don't mix. Overrides --cache-dir.
• --cache-dir DIR — Directory the cache path derives from; for verify, the directory its <bib-dir>[-<bib-stem>]/<source>/ output goes under. Also CITEFINDER_CACHE_DIR in the env or cache_dir in a project or user config. For per-project caches, set it once in the project config rather than passing a flag on every command — see Configuration.
• --rows N (search only) — Number of results to return. Default 3.
• --mailto EMAIL — Opts the request into the source's polite pool (both OpenAlex and Crossref honor it): faster responses and a higher quota. Sent as a ?mailto=… query param; stripped from the cache key, so rotating the email doesn't invalidate prior entries. Also OPENALEX_MAILTO / CROSSREF_MAILTO in the env or mailto in config.toml; verify reads the one for whichever --source it runs against.
• --api-key KEY (OpenAlex only) — OpenAlex API key for higher rate limits and tier-specific endpoints. Also read from OPENALEX_API_KEY in the env or a .env file (loaded from cwd or any parent). Sent as Authorization: Bearer <key> so it never lands in cache keys, URL logs, or referer headers.
• --max-retries N — Retries after a 429/502/503/504 response; 0 disables. Default 3. Also OPENALEX_MAX_RETRIES / CROSSREF_MAX_RETRIES in the env or max_retries in config.toml.
• --min-interval SECONDS — Minimum gap between consecutive requests. Default 0.1 for OpenAlex; for Crossref 1.0, or 0.34 with a mailto set; 1.0 for Open Library and the Library of Congress. Also OPENALEX_MIN_INTERVAL / CROSSREF_MIN_INTERVAL in the env or min_interval in config.toml. verify reads the variables for whichever --source it runs against.
• --books (verify only) — Check book entries with an isbn against Open Library and the Library of Congress; see above. The two clients read OPENLIBRARY_* / LOC_* from the env or [openlibrary] / [loc] in config.toml; isbn takes them as --mailto (the contact Open Library asks for, sent in the User-Agent), --max-retries, and --min-interval, applied to both, plus --no-loc to stop at Open Library.

Claude Code skill

The use-citefinder Claude Code skill lives inside the package, under citefinder/prompts/skills/use-citefinder/, laid out per the Agent Skills specification: a SKILL.md and a references/ directory. It is never copied out. The instructions are printed on demand:

citefinder skill                              # the full skill body, from the installed package
citefinder skill use-citefinder               # ...the same, by name
citefinder skill --list                       # the skill bodies this package ships
citefinder doc --list                         # reference docs the body loads mid-step
citefinder doc use-citefinder/given-names     # ...printed the same way

The packaging, stub rendering, and drift check come from pkgskills; citefinder/host.py declares what citefinder ships.

citefinder install materializes only a ~1.7 KB stub into a .claude/ tree — the skill's frontmatter (the description triggers Claude Code matches on, which have to be readable off disk) plus a pointer to citefinder skill use-citefinder:

citefinder install                  # ~/.claude/ — serves every repo
citefinder install --local          # <repo>/.claude/ — vendored per repo
citefinder install --check          # one status row per stub (exits 1 unless ok)
citefinder install --force          # overwrite a file citefinder didn't generate

Because the body is fetched rather than copied, the instructions an agent reads always come from the version of citefinder that is actually installed. The failure this design removes was a hand-copied skill still documenting a pre-0.4 CLI in a repo whose lockfile pinned 0.4.2 — with no copy, there is nothing to fall out of date.

The stub carries a stamp naming the version and mode it was rendered for:

<!-- generated by citefinder 0.12.0 via pkgskills 0.5.1 (mode=global); do not edit. Regenerate with: citefinder install --force -->

The stub changes rarely, but it is still checkable — --check compares it against what the currently installed citefinder would render (ignoring both stamped versions, so a routine upgrade of citefinder or pkgskills alone never reads as drift), prints one row per stub with a reason on anything not ok, and exits non-zero unless every row is ok:

uv add -U citefinder          # or: uv tool upgrade citefinder
citefinder install --check
citefinder install --force    # re-materialize if it reports drift

Add --local to both commands when the stub is vendored in a repo. A file at the target path that citefinder did not generate (no stamp) is never overwritten without --force, so a hand-authored skill of the same name is safe. A stub written by a release before citefinder adopted pkgskills reports foreign; install --force replaces it. To change the skill's content, edit citefinder/prompts/skills/use-citefinder/ and release — there is no .claude/ copy to re-sync.

Cache maintenance

Two layouts write under cache_dir: the shared per-source cache (<cache-dir>/<source>.jsonl, used by doi, search, and the crossref subcommands) and one cache per verify run (<cache-dir>/<bib-dir>[-<bib-stem>]/<source>/<source>.jsonl, so each run's evidence sits beside its own results.json). Each verify run reads the shared cache as a read-only fallback: a miss in its own cache that the shared file can answer — a record or a cached 404 — is a hit, not a request. The run's header says which files it read:

Cache: data/citefinder/paper/openalex/openalex.jsonl (12 entries pre-loaded)
Fallback: data/citefinder/openalex.jsonl (1268 entries)

The pre-load count spans both files. A Fallback line showing 0 entries against a populated cache directory means the merge below has not run yet, or that a different cache_dir resolved. --no-fallback pins a run to its own file.

A run never writes to the shared cache, so what a fresh run fetches reaches later runs only after a merge, and a 404 cached before a deposit landed lives on next to the record that resolved it months later. The cache commands consolidate and inspect them. Nothing else in the package merges anything — no lookup and no verify run consolidates as a side effect — so a misdirected row is never laundered into the shared cache by routine use.

citefinder cache stats                                   # inventory every cache under cache_dir
citefinder cache merge                                   # dry run
citefinder cache merge --write                           # rewrite <cache-dir>/<source>.jsonl
citefinder cache merge --write --keep-records            # ...never letting a 404 replace a record
citefinder cache merge --extra ~/backup/openalex.jsonl   # fold in a file from elsewhere
citefinder cache compact data/citefinder/openalex.jsonl  # dedupe one file, in place

How a merge decides:

• Newest ts wins across files. Two caches are independent logs, so the order they are read in says nothing; within one file the later line wins, as the cache itself replays it. The winner keeps its own ts — the fetch time is the only freshness signal a row carries — and a row with no ts at all loses every contest rather than being dropped.
• Rows are routed by the host in their key, never by the file name, so a record that landed in the wrong file is filed under the source that actually answered it (reported as misrouted rows).
• A newer 404 replaces a record, and the count is reported every run. --keep-records makes that never happen, for when a transient upstream failure is the likelier explanation than a real retraction.
• Inputs are never modified. merge only rewrites <cache-dir>/<source>.jsonl, so every per-run cache stays intact and the merge is repeatable. compact rewrites the one file it is given, and keeps rows from neither API rather than dropping them.
• Writes are atomic: a temporary file moved over the target, never an in-place rewrite that could race an appender. Both commands are a dry run until --write.

Do not merge while a lookup or verify run is writing into the same directory: the merge reads a moving target, and rows appended after it started are simply not in the output. Re-running it afterwards picks them up.

Why JSONL?

The cache is an append-only log: every lookup is one JSON object per line. Benefits:

• Auditable: cat/grep to see every query that ever ran.
• Diffable: plays nicely with git if you want to commit a project's cache.
• Crash-safe: an interrupted write loses at most the last line.
• Recoverable: rebuild the in-memory dict by replaying the log.

Latest value wins on replay, so over-writes are a no-op semantic.

SQLite alternative. A SQLite-backed cache is another reasonable implementation — it would trade the audit log and grep-ability for faster random access on very large caches (millions of entries) and concurrent writers. The current scale of citefinder use (per-project bibs, tens of thousands of entries at most) doesn't need it, and replaying a JSONL on startup is fast enough that the simplicity wins. If a future workload pushes past those limits, swapping the storage layer is a single class — JsonlCache in citefinder/cache.py — behind the same get / put / __contains__ interface.

Tests

uv run pytest

Changelog

See CHANGELOG.md for the release history.