rustypaper

Structure-aware conversion of born-digital scientific PDFs to Markdown, Typst, JSON and plain text. Headings, reading order, figures, tables, equations and references, on a CPU, with no models and no native libraries.

The good open-source converters (Marker, MinerU, Docling, Nougat) are Python stacks that want a GPU; GROBID is CPU-only and fast but is a JVM service that emits TEI and ignores maths. The Rust crates that exist are generic text extractors with no notion of a paper. This aims at the gap: structure-aware, maths-aware, CPU-only, single binary.

Status: feature complete. Converts one- and two-column papers to Markdown, Typst, plain text or JSON, with reading order, figures, tables, mathematics and references.

Install

pip install rustypaper
rustypaper convert paper.pdf
python -m rustypaper convert paper.pdf

The Python installation includes the same command line interface as the Rust executable. All commands and options are shared, including batch conversion, figure extraction, and diagnostics. No Rust compiler is needed when installing a supported platform wheel.

Standalone executables are available on the GitHub Releases page. Download the archive for your platform, verify it against SHA256SUMS, extract it, and put rustypaper (or rustypaper.exe) on your PATH. These executables do not need Python.

Alternatively, use cargo-binstall to download a release executable, or Cargo to compile it locally:

cargo binstall rustypaper
cargo install rustypaper

For the Rust library:

rustypaper = "0.2"

Nothing else to install. PDFs are read with rustium-pdf, a pure-Rust interpreter, so there is no native library to fetch, point an environment variable at, or match versions with. ldd on the binary shows libc, libm and libgcc and nothing else; the wheel is the extension module and the Python package around it, with no C library travelling beside it.

Getting started

scripts/fetch-corpus.sh     # evaluation corpus of arXiv papers, not committed
scripts/build.sh            # cargo build --release, plus installing the Python extension

# Convert.
./target/release/rustypaper convert corpus/resnet.pdf
./target/release/rustypaper convert corpus/resnet.pdf --format typst --assets figures/
./target/release/rustypaper convert corpus/*.pdf --out out/        # batch
./target/release/rustypaper convert paper.pdf --caveman=hard       # -24% words for LLM ingestion

# Diagnostics.
./target/release/rustypaper probe corpus/resnet.pdf --pages   # counts, fonts, detected gutters
./target/release/rustypaper text  corpus/resnet.pdf --geometry # reconstructed lines
./target/release/rustypaper dump  corpus/resnet.pdf --page 0 --pretty

probe prints per-page counts, the font histogram and the detected gutters. Gutters are the first thing to check when a two-column paper comes out interleaved; the font histogram is the first thing to check when text comes out wrong.

Design

Read docs/ARCHITECTURE.md for the pipeline, the coordinate convention, the PageSource boundary, and the things the corpus taught this converter the hard way.

The short version: the backend produces a PageRaw of glyphs, paths and images, and every later stage is a pass over an IR. The Document JSON is the contract; Markdown, Typst and plain text are renderings of it.

Unicode repair turned out not to be needed — glyph-name fallback already resolves TeX ligatures, so the tables the plan budgeted for were dropped. De-hyphenation needed a second mechanism the plan did not anticipate: the page is read as written, so a soft line-break hyphen is usually there and is preferred when it is, but where a document leaves none the words split across the break are rejoined using the document's own vocabulary, which needs no word list and knows the paper's jargon.

Maths is reconstructed geometrically, MaxTract-style, from exact glyph identities and positions rather than by OCR — a born-digital PDF hands you perfect character information, so image-to- LaTeX models are solving a problem this pipeline does not have. Equations carry a confidence score and fall back to a rendered crop rather than emitting confident-looking nonsense.

Scanned documents are explicitly out of scope: extract fails with Error::Scanned rather than pretending.

Milestones

Python

The core is Rust; the tooling around it is Python, because evaluation, corpus management and comparison against other converters are scripting jobs.

scripts/build.sh
PYTHONPATH=python python3 -c "
import rustypaper
print(rustypaper.to_markdown('corpus/resnet.pdf')[:80])
doc = rustypaper.to_document('corpus/resnet.pdf')   # the document model as a dict
for section in doc['sections']:                     # the outline, with block ranges
    print(section['level'], section['title'], section['start'], section['end'])

markdown, doc = rustypaper.convert('corpus/resnet.pdf')   # both, from one pipeline run
print(rustypaper.to_typst('corpus/resnet.pdf')[:80])
print(rustypaper.to_text('corpus/resnet.pdf')[:80])
"

sections is the document's outline: each entry has a title (None for the front matter that precedes every heading), a level, a half-open start/end range into blocks that includes its nested children, and the pages it spans. A consumer that wants the methods section can slice the blocks rather than re-deriving structure from the Markdown.

ScannedDocument is raised for image-only PDFs, so callers can route those to an OCR pipeline instead. Conversion releases the GIL, so several threads convert in parallel.

Evaluation

Quality is measured, not eyeballed. Papers submitted to arXiv as TeX source come with the prose their PDF was rendered from, which is free ground truth for exactly this document class — for the subset of papers that have it. PDF-only submissions have none, and are reported as skipped rather than scored.

cd eval && PYTHONPATH=.:../python python3 -m rustypaper_eval

Current scores across the fifteen scorable papers:

The maths numbers are the honest state of the differentiator, and they are the project's weakest point — see docs/ARCHITECTURE.md. See eval/README.md for what the metrics mean and why plain edit distance is the wrong primary measure here.

Testing

cargo test                                    # unit, robustness and corpus tests; corpus tests
                                              # skip if corpus/ is empty
python3 -m unittest discover -s eval/tests    # the eval harness's own tests
PYTHONPATH=python pytest python/tests -q      # the Python surface, against the corpus
cd eval && PYTHONPATH=.:../python python3 -m rustypaper_eval --baseline baseline.json

Integration tests live in rustypaper/tests/corpus.rs and run against real papers. They skip rather than fail when the corpus is absent, so a fresh clone is green.

Licence

MIT OR Apache-2.0.

That covers everything a build contains, the published crate and the published wheels included: the dependency tree is Rust, and rustium-pdf is MIT OR Apache-2.0 as well.