Extra citations support for FLM

See the FLM README file.

Install with:

$ pip install flm-citations

Use the additional config front matter in your FLM files to enable citations with automatic citation retrieval from arXiv, DOI, etc.

---
$import:
  -  pkg:flm_citations
bibliography:
  - my-csl-bibliography.yaml
---

Then process your file as usual with flm.

The bibliography file(s) you provide (in the example above, my-csl-bibliography.yaml) should be in CSL JSON or CSL YAML format. They can easily be exported from Zotero, for example.

With the default configuration, the following citation keys are processed:

• \cite{arXiv:XXXX.YYYYY} - fetch citation information from the arXiv, and from its corresponding DOI if applicable.
• \cite{doi:XXX} - fetch citation information using its DOI
• \cite{manual:{X et al., Journal of Future Results (2034)}} - manual citation text
• \cite{bib:BibKey2023} - use a citation from any of your bibliography files specified in your document front matter.

Configuration

All options go under flm.features.flm_citations in your config or front matter.

Progress is otherwise reported through the flm_citations.retrieve logger: at INFO you get passes and per-source counters, at DEBUG every HTTP request and resolved citation. Nothing is reported, and no reporter is even attached, when that logger is quieter than INFO.

Each entry of sources is {name: …, config: {…}}. A source's cite_prefix chooses the citation prefix it answers to, so the same kind of source can be registered more than once — for instance a second bibliography over different files:

flm:
  features:
    flm_citations:
      sources:
        - $defaults:
        - $merge-config:
            name: 'bibliographyfile'
            config:
              cite_prefix: b
              bibliography_file:
                - my-other-bibliography.yaml

Refresh batching

Cached entries expire (arXiv after 10 days, DOIs after 360), and refreshing a handful of them costs as much rate-limited time as refreshing many. So by default a source waits until enough entries are stale to be worth a request — arXiv 20, doi.org 10 — and tops a request that is going out anyway up with entries that are nearly due. Missing citations are always fetched straight away, and a stale entry is served from the cache while it waits (at most 2 days past expiry for arXiv, 30 for DOIs). A bibliography file is re-read in full whenever any of its entries is refreshed.

A source's refresh_batching option adjusts this:

sources:
  - name: arxiv
    config:
      refresh_batching:
        min_batch: 50          # wait for 50 stale entries...
        max_defer_days: 7      # ...but never more than a week past expiry
        top_up:
          min_age_percent: 50  # fill requests with entries half-way through their lifetime
          fill: chunk          # up to the request's capacity (or a count, e.g. 5)
  - name: doi
    config:
      refresh_batching: eager  # refetch whatever is due, as soon as it is due

Settings not given keep the source's default; eager: true inside the mapping starts from eager instead, and top_up: false never pulls entries forward.

The citation cache

Resolved citations are cached in .flm-citations.jsonl next to your document: one sorted entry per line, so a diff stays readable. Commit it if you want builds to be reproducible offline. A run also creates a few throwaway lock and log files while it is in flight; they all carry a ._ prefix and delete themselves, so one ignore rule covers them:

._.flm-citations*

Manual citation text is never written to the cache at all, so arbitrary \cite{manual:…} text cannot leak into a committed file.

Metadata Fetching

Thank you to arXiv and doi.org for use of their open access interoperability.

Since version 0.3, retrieval is handled by autocitefetch, a Rust library shipped as a compiled extension module. It brings automatic retry with backoff, correct rate limiting, stale-while-revalidate caching, and per-citation error tolerance. This package keeps the FLM integration and the citeproc-py rendering.

In case citeproc chokes on certain entries fetched by DOI

Sometimes automatically generated citeproc/JSON entries fetched through various available online APIs (doi.org, crossref.org, arXiv.org, etc.) might not be fully conforming or exactly matching the structure expected by the citeproc-py citation formatting library that this project uses. If you run against such issues, you might consider installing a patched version of the library that smoothed out some issues I had in the past; you can install it with

> pip install git+https://github.com/phfaist/citeproc-py.git@pr-branch

until my upstream PR is considered.

Note that citeproc-py needs issued in the {"date-parts": [[2022, 8, 12]]} form; the [{year: 2022, month: 8}] shape some CSL-YAML exporters produce is not accepted.

Upgrading from 0.2

Existing configurations keep working, with two exceptions and a few deprecations that are reported when they are used:

• Custom Python citation source classes are gone. A sources entry naming an importable class (name: 'my.module.MyClass') is now an error; retrieval happens in the Rust library, which has no Python source interface.
• cache_file is replaced by cache_dir + cache_base. The cache is no longer a single JSON file. The old .flm-citations.cache.json is not read and can be deleted.
• cache_entry_duration_dt is ignored. Cache lifetimes are now per source (arXiv 10 days, DOI 360 days, bibliography files 60 s, manual entries not cached); only a bibliography source's lifetime is configurable, via ttl_seconds.
• chunk_size and chunk_query_delay_ms are ignored. Chunk sizes and request pacing are properties of each remote API and are fixed by the library.

Two output differences are worth knowing about:

• An arXiv entry's date is now its last revision (<updated>) rather than its original submission, so a paper's citation year may change.
• write_csljson_file / write_bibtex_file now export exactly the citations the document uses, rather than everything that happened to be in the cache.

Development

pip install maturin
maturin develop            # build the extension into the current virtualenv
cargo test                 # Rust unit tests
pytest -m 'not network'    # Python tests, offline
pytest -m network          # live arXiv/doi.org checks (slow, rate-limited)

The Rust half depends on autocitefetch, pulled as a git dependency over anonymous https; no credentials are needed to build.

License

MIT