pyferro-pdf 0.1.0

Fast, layout-aware PDF text extraction in Rust with Python bindings

ferro-pdf

Fast, layout-aware PDF text extraction written in Rust, with Python bindings.

Most PDF text extractors are either slow (pdfminer.six is pure Python) or flat (they emit glyphs in content-stream order, which scrambles multi-column pages). ferro-pdf is built for RAG / document-ingestion pipelines: it reconstructs reading order (columns, lines, spacing) and is fast enough to not be the bottleneck.

Benchmark

Synthetic corpus (clean, standard fonts)

9 generated PDFs, 198 pages, single- and two-column. Machine: Apple M4 Pro (14 cores), macOS 26.5. Best-of-5, warm cache.

tool	best (s)	speedup vs pdfminer	content (words vs pdfminer)
pdfminer.six	2.68	1.0×	100%
pypdfium2 (C++)	0.098	27.3×	100%
ferro-pdf	0.019	138.6×	100%

Real research papers (arXiv, embedded fonts, math, two-column)

6 papers (Attention, BERT, GPT-3, InstructGPT, LLaMA, ResNet), 213 pages total.

tool	best (s)	speedup vs pdfminer	content (words vs pdfminer)
pdfminer.six	4.33	1.0×	100%
pypdfium2 (C++)	0.224	19.4×	100%
ferro-pdf	0.135	32.1×	98%

Per-paper: 19–43× faster, 91–100% word parity. The parity gap on dense two-column papers (ResNet 91%, BERT 95%) comes from math/figure/table content where word tokenization diverges from pdfminer — the next thing to harden.

Update: since this table was recorded, real-paper word parity improved to ~99% after implementing full font-encoding support (Identity-H CID, /Differences) and real-glyph-width spacing. Re-run bench.py papers to refresh these numbers on your machine.

Takeaway: on clean PDFs the win is ~140×; on messy real-world papers it's a still-excellent ~30× (and ~1.7× faster than C++ pypdfium2), at 98% content parity. Quote the real-paper number publicly — it survives scrutiny.

Reproduce:

python gen_corpus.py corpus      # synthetic PDFs
python bench.py corpus           # synthetic benchmark
python bench.py papers           # real-paper benchmark (after downloading PDFs)

The speedup is content-preserving: ferro-pdf extracts ~the same word count as pdfminer (the reference), so the win isn't from extracting less text.

How it works

PDF bytes
  │  lopdf            parse objects + decode content streams (don't reinvent this)
  ▼
glyph extraction      walk content operators, track CTM/text-matrix, decode
  │  (src/extract.rs)  Unicode via ToUnicode CMap, Identity-H CID, or simple-font
  │                    /Encoding + /Differences; measure real glyph widths
  ▼
layout engine ★       column detection (x-coverage histogram + gutter finding),
  │  (src/layout.rs)   line bucketing by y, gap-based spacing, reading order
  ▼
rayon                 lay out pages in parallel (pdfminer is single-threaded)
  ▼
clean reading-order text

The layout engine is the moat — it's what flat Rust crates (pdf-extract) and even C++ pypdfium2 don't do for you.

Usage

Python

pip install pyferro-pdf

import ferro_pdf
text = ferro_pdf.extract_text("paper.pdf")
text = ferro_pdf.extract_text_from_bytes(open("paper.pdf", "rb").read())

Rust

let text = ferro_pdf::extract_text_from_path("paper.pdf")?;

CLI

cargo build --release
./target/release/ferro-cli paper.pdf            # print text
./target/release/ferro-cli paper.pdf --time     # print timing to stderr

Build

# Rust library + CLI
cargo build --release

# Python wheel (into the active venv)
maturin develop --release

Honest limitations (current PoC)

• Text-only: table reconstruction (ruling lines → Markdown grid) is not implemented yet — that's the next moat to build.
• Scanned / image-only PDFs need an OCR fallback (not included).
• Column detection is heuristic (gutter finding); pathological layouts may need tuning of the thresholds in detect_columns.
• Font decoding handles ToUnicode CMaps (incl. array-form bfrange), Identity-H CID fonts, and simple fonts via WinAnsi/MacRoman + /Differences. Truly exotic encodings without any of these signals may still mis-decode.

These are the honest gaps to close before claiming production parity — but the core speed + reading-order story is real and reproducible above.

Performance: where the time goes & further scope

Stage profile (FERRO_PROFILE=1, GPT-3, 75 pages), after optimization:

stage	time	notes
load	~42 ms (~85%)	lopdf parse + inflate — the bottleneck, single-threaded
extract	~4 ms	content-stream + font decode (was ~31 ms before the font cache)
layout	~3 ms	column/line reconstruction — negligible

Optimizations already applied:

• Font/ToUnicode cache (parse each CMap once, not per page): ~7× on the extract stage (GPT-3 31 ms → 4 ms).
• Parallel page extraction + parallel batch API (extract_text_batch): 9.2× scaling on 14 cores.

Throughput (60-PDF / 2,130-page batch, Apple M4 Pro):

mode	time	vs pdfminer	notes
pdfminer.six (sequential)	43.9 s	1.0×
pypdfium2 — 1 core	2.32 s	18.9×	C++; fastest single-core
pypdfium2 — threads	crashes	—	pdfium is not thread-safe
pypdfium2 — process pool	1.42 s	31×	fork + IPC overhead
ferro — 1 core	5.63 s	7.8×	lopdf parse/inflate-bound
ferro — parallel batch	0.59 s	74×	no-GIL Rust threads

Honest read vs pypdfium2: per single core, pdfium (C++) is ~2.4× faster than ferro — it's heavily optimized and ferro is bound by lopdf. ferro wins at batch scale (2.4× faster than pdfium's process pool) because pdfium can't use threads, while ferro parallelizes freely with no GIL. ferro's edge is parallelism + pure-Rust embeddability + reading-order layout output, not raw single-core speed.

Remaining scope, ranked by expected payoff:

1. Attack load (85% of single-core time). This is also exactly the gap to pypdfium2. SIMD inflate (flate2 zlib-ng, needs cmake) ≈ 2–3× on decompression; skip image/XObject streams we never use; or a lazier parser. Closing this would make ferro competitive with pdfium single-core too.
2. Parallel granularity. 9.2× / 14 cores ≈ 66% efficiency; file-level-only parallelism for batch (avoid nested rayon overhead) could reach ~12×.
3. Allocation trimming in the operator loop (arena / reuse buffers): modest.

Beyond ~2× more you'd need to drop lopdf for an FFI to pdfium/mupdf — at which point you lose the "pure Rust" story and pdfium would roughly match you.