litdown

Convert scholarly full-text XML to Markdown with embedded LaTeX for inline and display math. Two dialects are supported behind a single convert entry point that sniffs the document root and dispatches:

• JATS — a journal article (<article>, the format PubMed Central distributes) or a BITS book part (<book-part-wrapper>, one NCBI Bookshelf chapter as Europe PMC's bookXML endpoint serves it; a GeneReviews entry, say). The chapter's <book-part-meta> stands in for <front>.
• Elsevier (<full-text-retrieval-response>) — the ScienceDirect Article Retrieval API's xocs/ja/ce schema.

The intended consumer is downstream LLM tooling — the markdown is plain text suitable for retrieval, summarisation, or analysis without round-tripping through a typesetter.

Spec target

The JATS dialect is implemented against the JATS Journal Archiving and Interchange Tag Set (Archiving), NISO Z39.96-2024 v1.4 — the format PMC distributes. PMC upconverts older content (NLM Archiving 1.x–3.x, JATS 1.0–1.3) into 1.4 when serving the OA bucket, so a converter that handles 1.4 covers the entire PMC corpus regardless of when the article was authored.

This is not the Article Authoring tag set (more restrictive; intended as an authoring target, not a corpus). Article-Authoring-only content is a subset of Archiving content and works without code changes.

BITS (Book Interchange Tag Suite 2.x; content models checked against the 2.1 Tag Library) is the book counterpart: a <book-part>'s <body> and <back> share the Archiving content model, so the same renderer handles them and reads the chapter's title and abstracts from <book-part-meta>. A <book-part-wrapper> holds one unit of a book, and only a <book-part> is rendered — a wrapper around a <book-app>, <preface>, <glossary> or other unit raises ValueError. A part whose body nests further parts renders them as sections, one heading level down per nesting.

The Elsevier dialect targets the ce:/ja:/xocs: schema returned by the ScienceDirect Article Retrieval API. Math is standard W3C MathML (shared with the JATS math path); tables are CALS (tgroup/row/entry); references parse the structured sb: (Siemens) model. An unrecognised root element raises ValueError rather than returning an empty string, so a caller passing the wrong bytes fails loudly.

Install

pip install -e .              # runtime
pip install -e '.[dev]'       # runtime + pytest
pip install -r requirements-dev.txt && pre-commit install   # contributing

Editable install. Provides a litdown console script.

Use

CLI:

litdown article.xml > article.md
litdown article.xml article.md

Library:

from litdown import convert, mml_to_tex, render_mathml

md = convert("article.xml")           # JATS (article or book part) or Elsevier → markdown
latex = mml_to_tex(math_element)      # MathML Element → LaTeX
fragment = render_mathml(math_element, display=True)  # → "$$...$$"

What's in the package

litdown/
  jats.py      JATS XML → Markdown
  elsevier.py  Elsevier (ce:/ja:/xocs:) XML → Markdown
  common.py    dialect-neutral leaves (tag helpers, table grid, inline wraps)
  mathml.py    MathML → LaTeX

The MathML converter is the more battle-tested piece — it has been graded against the W3C MathML 3 Presentation test suite using both Pandoc and a Gemini blind-grading harness. The cases that survived grading are checked in under tests/w3c_mml/ with their expected LaTeX in tests/golden.json; the regression suite re-runs the converter over them on every test run.

Tests and fixtures

pytest                                # full suite

Seven test files:

• tests/test_mml_unit.py — exhaustive per-element MathML cases.
• tests/test_jats_articles.py — structural assertions over real PMC articles in tests/fixtures/<PMCID>/, parametrised so adding a fixture extends the suite automatically. Known per-fixture defects are xfail-marked in a KNOWN_BUGS dict so the suite stays green; when a fix lands the xfail flips to "unexpectedly passed" and forces the entry's removal.
• tests/test_elsevier_articles.py — structural assertions over Elsevier articles committed as flat *.xml files under tests/fixtures/elsevier/ (math not dropped, CALS tables rendered, every cross-ref/float/reference anchored). Vendor only CC-BY (by/4.0) articles; see docs/elsevier-dialect-plan.md for how to harvest fixtures.
• tests/test_book_part_unit.py — hand-written BITS <book-part-wrapper> documents (Bookshelf prose is not redistributable): the title group, abstracts, nested parts, refusal of other wrapper units, reference labels, and a <ref-list> inside a <sec>.
• tests/test_convert_inputs.py — convert accepts a path, raw bytes or a binary stream and yields the same Markdown, for both dialects.
• tests/test_whitespace_unit.py — single-line Markdown constructs (headings, table rows, list items) survive pretty-printed source, one hand-written document per construct.
• tests/test_block_content_unit.py — block content inside a container (a nested list, a second paragraph or a definition list under a list item; lists in table cells, footnotes, boxed text and quotes) survives in both dialects and nests as Markdown; one hand-written document per shape.

Fetching test fixtures

PMC articles are not redistributed in this repository — each article has its own licence (a mix of CC-BY, CC-BY-NC variants, and others), and the publisher PDFs in particular carry more restrictive terms. The fixture directories are gitignored. To populate them:

python tools/fetch_pmc.py --manifest tests/fixtures/MANIFEST.txt

This reads tests/fixtures/MANIFEST.txt (one PMCID per line), pulls each article's JATS XML, publisher PDF, plain text, and referenced figure assets from the public pmc-oa-opendata S3 bucket, and caches them under tests/fixtures/<PMCID>/. Fetches are idempotent; re-running is cheap.

The article-fixture tests skip cleanly when no fixtures are present, so pytest works against the MathML unit suite alone.

tools/

Discovery and evaluation utilities — none are imported by the package or needed for normal use.

The discovery loop

        fetch_pmc.py            (acquire fixture)
              ↓
        litdown.convert
              ↓
        eval_articles.py        (Gemini reads PDF + our markdown)
              ↓
        triage findings         → encode each as a structural test
              ↓                    → fix the converter
        re-run, repeat

The structural test suite is the regression net (deterministic, runs in CI). LLM eval is the discovery tool (non-deterministic, runs ad-hoc). Each real defect the eval surfaces should be added to tests/test_jats_articles.py once fixed, so it can never silently regress.

Known limitations

• Tables typeset as images (older PLOS Genetics, BMJ, etc.) cannot be reconstructed as markdown tables — the converter falls back to an image link so content isn't lost, but downstream tools won't get structured data without an OCR step.
• The consortium author rendering for papers like gnomAD (PMC7334197) emits the consortium name only; individual members listed in nested <contrib-group> are dropped.
• Soft hyphens / line-break artefacts in source XML are not normalised, so words split across lines in the JATS source can render with stray spaces ("si milarity").
• A BITS <book-part-meta> yields the labelled title, its subtitles and the abstracts only; contributors, <alt-title>, translated titles, keyword groups, funding and notes are not rendered, nor are the enclosing <book-meta> and a part's own <front-matter>.