razorpdf — Hybrid Geometric PDF Parser for RAG & MCP

Ultra-fast, local-first PDF parser: PDFs in, clean LLM-ready Markdown plus grounded RAG chunks out.

Deterministic Rust geometry handles the easy 90%+ of document layouts in microseconds per page (16,720 pages/sec on synthetic benchmarks; 339 pages/sec across real arXiv papers and tax invoices). Only genuinely messy sub-page regions ($S(R) \ge 0.45$) ever get cropped to WebP and sent to a Vision Language Model (VLM).

Try it in 30 seconds

pip install razorpdf          # Python 3.10+, prebuilt wheels, no Rust needed
cargo install razorpdf        # CLI + MCP server from source

import razorpdf
doc = razorpdf.parse_file("paper.pdf")   # -> {"markdown": ..., "chunks": [...]}

Prefer a binary? Grab one from Releases (Linux shown):

curl -L -o razorpdf.tar.gz https://github.com/shridhargsabat/razorpdf/releases/download/v0.1.6/razorpdf-v0.1.6-x86_64-unknown-linux-gnu.tar.gz
tar xzf razorpdf.tar.gz && ./razorpdf parse paper.pdf -o paper.md --json paper.json

Or run the container (builds once, then parse + MCP server included):

git clone https://github.com/shridhargsabat/razorpdf.git && cd razorpdf
docker compose run --rm parse   # samples/invoice.pdf -> ./out/invoice.{md,json}

Prefer a prebuilt image? Each release publishes to GHCR:

docker run --rm -v ./samples:/work:ro ghcr.io/shridhargsabat/razorpdf:latest \
  parse /work/invoice.pdf

No Docker either? Try it in your browser:

Why razorpdf?

Most PDF pipelines force an expensive tradeoff: flat text extractors mangle multi-column papers and borderless tables, while full-page vision/OCR pipelines burn seconds and thousands of image tokens on simple text pages. razorpdf combines both worlds:

Why sub-page VLM wins on cost

Full-page vision pipelines pay image tokens for every page. razorpdf pays zero for geometric pages and only a small crop for the rest — measured on the wild suite:

At typical frontier-VLM image pricing (~$2.50/1M input tokens, illustrative), that is ~$0.0053 per full-res page vs ~$0.0011 per crop — and geometry itself runs in ~60 µs/page with no model at all.

What You Get Out

Pass any PDF via CLI, Python, or MCP and receive both structured Markdown and grounded RAG chunks with exact bounding boxes, heading hierarchies, token counts, and provenance flags:

{
  "total_pages": 15,
  "markdown": "# Attention Is All You Need\n\n...",
  "chunks": [
    {
      "chunk_id": "p5-005",
      "page_index": 5,
      "bbox": {"x0": 124.5, "y0": 137.2, "x1": 468.0, "y1": 181.4},
      "breadcrumb": ["Attention Is All You Need", "3 Model Architecture"],
      "markdown": "| Self-Attention | O(n2 ·d) | O(1) | O(1) |\n| --- | ...",
      "token_estimate": 57,
      "parsed_by_vlm": false,
      "page_spans": [{"page": 5, "bbox": {"x0": 124.5, "y0": 137.2, "x1": 468.0, "y1": 181.4}}]
    }
  ]
}

Architecture

PDF bytes
    │
    ▼
┌─────────┐   lopdf (pure Rust, rayon page-parallel)
│ EXTRACT │─── TextSpan + VectorSegment + image bboxes (top-left coords)
└─────────┘
    │
    ▼
┌──────────┐  line merge -> XY-cut reading order -> vector table grids
│ GEOMETRY │── multi-column safe, zero model calls
└──────────┘
    │
    ▼
┌────────┐  S(R) = min(1, 0.40·poverlap + 0.35·ptables + 0.25·pvectors)
│ TRIAGE │── fast path (< 0.45) or padded sub-page VLM region
└────────┘
    │
    │ WebP crop of BBox only (never full pages)
    ▼
┌─────┐  OpenAI / Gemini / Ollama / any OpenAI-compatible endpoint
│ VLM │── bounded concurrency, per-region failure tolerance
└─────┘
    │  │
    ▼  ▼
┌──────────┐ reading-order Markdown + chunks with
│ ASSEMBLE │── `{x0,y0,x1,y1}` bboxes + breadcrumb trails
└──────────┘

• Fast path is pure geometry: Line merging, recursive XY-cut, and R-tree vector-table reconstruction run in parallel across cores with zero model calls.
• VLM fallback is sub-page: Only the failing BBox is cropped to WebP and sent — a messy borderless table costs one small image crop, not a full-page render.
• Chunks are grounded: Every chunk carries its chunk_id, page_index, bbox ({x0, y0, x1, y1} in PDF points), heading breadcrumb trail, token_estimate, the parsed_by_vlm provenance flag, and multi-page page_spans for stitched content.
• Offline by default: No network calls run in tests or on the fast path. Without --vlm-url or --mock-vlm, messy regions keep their raw geometric text so content is never dropped silently.

Quickstart

1. CLI (Rust)

cargo install razorpdf

# Parse a PDF to Markdown (stdout or file) plus grounded JSON chunks
razorpdf parse paper.pdf -o paper.md --json paper.json

# Inspect per-page spans, vectors, lines, tables, S(R) triage scores, and fallback regions
razorpdf inspect paper.pdf

# Route messy sub-page regions to local Ollama (or any OpenAI-compatible VLM endpoint)
razorpdf parse scan.pdf --vlm-url http://localhost:11434/v1/chat/completions

# Run an offline demo of the VLM fallback path using the deterministic mock provider
razorpdf parse messy.pdf --mock-vlm

# Structured extraction, revision diffs, and JSONL chunk streams
razorpdf schema invoice.pdf --schema '{"type":"object","properties":{"total":{"type":"number"}}}'
razorpdf diff old.pdf new.pdf
razorpdf jsonl paper.pdf -o chunks.jsonl

From source: git clone https://github.com/shridhargsabat/razorpdf.git && cd razorpdf && cargo build --release -p razorpdf.

2. Python Bindings

razorpdf provides native Rust bindings via PyO3/maturin and releases the Python GIL during parsing so thread pools scale cleanly across CPU cores.

pip install razorpdf   # prebuilt abi3 wheels for Linux/macOS/Windows, Python 3.10+

import razorpdf

# Full grounded document (dict with 'markdown' and 'chunks')
doc = razorpdf.parse_file("paper.pdf")
print(doc["markdown"][:500])
print(doc["chunks"][0]["bbox"], doc["chunks"][0]["breadcrumb"])

# Fast Markdown-only extraction (str)
md = razorpdf.to_markdown("paper.pdf")

# Route high-entropy regions to a local or cloud OpenAI-compatible VLM
doc = razorpdf.parse_file(
    "scan.pdf",
    vlm_url="http://localhost:11434/v1/chat/completions"
)

# Options: stitch_pages=True joins tables/paragraphs split across pages;
# use_cache=True reuses blake3-keyed VLM responses across runs
doc = razorpdf.parse_file("long.pdf", stitch_pages=True, use_cache=True)

# BM25 + phrase retrieval over grounded chunks (offline, no VLM)
hits = razorpdf.search_chunks("paper.pdf", "attention", top_k=5)
print(hits[0]["chunk_id"], hits[0]["page"], hits[0]["score"])

# Highlighted citation crop of any region as WebP bytes (page is 1-based)
webp = razorpdf.render_citation("paper.pdf", 1, [72.0, 100.0, 540.0, 200.0])
open("cite.webp", "wb").write(webp)

# Structured extraction into any JSON Schema (per-field bbox provenance)
import json
schema = json.dumps({"type": "object", "properties": {"total": {"type": "number"}}})
result = razorpdf.extract_schema("invoice.pdf", schema)
print(result["data"], result["overall_confidence"])

# Revision diff, columnar export, and JSONL streaming
diff = razorpdf.diff_pdfs("old.pdf", "new.pdf")
batch = razorpdf.to_columnar_dict("paper.pdf")  # polars.from_dict(batch)
jsonl = razorpdf.to_jsonl("paper.pdf")

From source: cd crates/razorpdf-py && pip install maturin && maturin develop --release.

3. MCP Server (Claude Code / OpenCode / Hermes)

razorpdf includes a built-in Model Context Protocol (MCP) JSON-RPC 2.0 server over stdio exposing ten tools: parse_pdf, inspect_pdf_layout, and extract_pdf_tables, plus progressive-disclosure tools get_pdf_toc (outline + per-page complexity scores), search_pdf_chunks (offline BM25 retrieval with bboxes), read_pdf_page_range (1-based page slices), render_pdf_region (WebP image blocks with citation highlights), extract_pdf_schema (structured JSON + field provenance), diff_pdfs (revision redlines), and extract_pdf_annotations_and_forms (links, form fields, footnotes). Parsed documents are cached in memory by path + mtime, so follow-up calls on the same PDF answer in milliseconds.

Claude Code:

claude mcp add razorpdf /path/to/razorpdf/target/release/razorpdf mcp

OpenCode (opencode.json):

{
  "mcp": {
    "razorpdf": {
      "type": "local",
      "command": ["/path/to/razorpdf/target/release/razorpdf", "mcp"]
    }
  }
}

Generic MCP / Hermes (stdio configuration):

{
  "name": "razorpdf",
  "transport": "stdio",
  "command": "/path/to/razorpdf/target/release/razorpdf",
  "args": ["mcp"]
}

Example round-trip over stdio:

echo '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"parse_pdf","arguments":{"path":"paper.pdf"}}}' | ./target/release/razorpdf mcp

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "{\"total_pages\":1,\"markdown\":\"# Sample Document\\n...\",\"chunks\":[{\"chunk_id\":\"p0-000\",\"page_index\":0,\"bbox\":{\"x0\":72.0,\"y0\":72.0,\"x1\":540.0,\"y1\":120.0},\"breadcrumb\":[\"Sample Document\"],\"parsed_by_vlm\":false}]}"
      }
    ]
  }
}

4. More ways to try

• Gradio demo / HF Space: demo/ is a Space-ready app (upload a PDF, get Markdown + chunks + bbox view) — see demo/README.md.
• Notebook: docs/notebook/razorpdf_quickstart.ipynb (Colab badge above).
• RAG frameworks: docs/integrations.md — LangChain loader (tested, in demo/) + LlamaIndex snippet.
• VLM providers: docs/vlm-providers.md — Ollama, OpenAI, Gemini, and any OpenAI-compatible endpoint.

CLI Reference

Examples

Bordered invoice table recovered exactly (green box), text around it untouched:

Rulings-free table under a rotated watermark — the watermark stays isolated in its own chunk and never merges into rows:

Machine-readable outputs live in examples/ (Markdown + grounded JSON for both invoices and Attention Table 1).

Benchmarks

Synthetic Fast-Path Throughput

Measured with razorpdf bench (400-page mixed two-column, single-column, and bordered-table paper, fast path, no VLM; median of 3 timed runs after 3 warmups) on an AMD EPYC 9354P (4 vCPU):

Reproduce: cargo run --release -p razorpdf -- bench Criterion suite: cargo bench -p razorpdf-core --bench throughput

Wild Head-to-Head (Real PDFs, Python API)

scripts/bench_wild.py benchmarks razorpdf.parse_file (release bindings) against flat extractors (pypdf, pdfminer.six, pymupdf) and fellow Rust parsers (rasgapdf, oxidize-pdf) on tests/wild/*.pdf (median of 3 runs after 1 warmup on the same EPYC 9354P machine; razorpdf 0.1.1, pypdf 6.21.4, pdfminer.six 20260107, pymupdf 2.0, rasgapdf 0.3.3, oxidize-pdf 0.23.1):

* rasgapdf 0.3.3 is genuinely the fastest flat extractor on the files it reads — but it returned empty output for both generated invoices (verified with path and bytes input), so its TOTAL is flattered by two instant empties. Compare per-file, and see fidelity/structure below.

Content Fidelity & Structure Check

Five backends recover the marker phrases, but only razorpdf reconstructs the tables as structured Markdown grids (the 3-row GST invoice grid and Attention's Tables 1–3 plus author block) rather than flat text.

Model Pipelines (Docling)

Heavy vision/layout pipelines trade latency for structure. On the two single-page files (docling 2.145.0, models cached, same machine):

Docling matches razorpdf on the GST invoice (5/5 markers, 3 table rows) — at ~3,400× the latency. That's the gap the hybrid design closes: geometry gives you the structure in microseconds, and the VLM bill only arrives for regions geometry can't solve.

Reproduce: .venv/bin/python scripts/bench_wild.py (after maturin develop --release). The corpus is gitignored — fetch it first per tests/wild/README.md; the script degrades gracefully on a partial corpus.

v0.2.0 Validation Run

Re-ran on release bindings (razorpdf 0.2.0, pypdf 6.21.4, pymupdf 2.1) on an Intel i5-13420H (12 threads), partial corpus (3 fetched PDFs, 115 pages; the 2 generated invoices were not regenerated). Median of 3 runs after 1 warmup:

Triage metrics from scripts/bench_omnidoc.py on the same corpus (offline, NoVlmProvider — flagged regions keep raw text): attention 3 tables / 573 chunks / 45 VLM regions (math-font clusters from the new MathGlyphCluster triage), deepseek 12 / 3,817 / 60, tracemonkey 0 / 890 / 32. samples/invoice.pdf: 2,200 p/s, 1 table, 6 chunks, 0 VLM regions. Fidelity 5/5 on Attention for all backends; structure 13 Markdown table rows for razorpdf, 0 for flat extractors.

OmniDocBench-style harness: scripts/bench_omnidoc.py [PDF_DIR] measures pages/sec, peak RSS, and table/chunk/VLM counts per file and writes reports/omnidoc_scorecard.md. With <stem>.json ground-truth sidecars ({"text": ..., "tables": [...]}), it also reports reading-order NED and table TEDS (Zhang-Shasha tree-edit similarity). scripts/bench_omnidoc.py --selftest validates the TEDS implementation without any PDFs.

Triage Math

For each candidate region $R$, crates/razorpdf-core/src/triage/scorer.rs computes a visual complexity score $S(R)$:

$$S(R) = \min(1.0, 0.40 \cdot p_{\text{overlap}} + 0.35 \cdot p_{\text{tab_entropy}} + 0.25 \cdot p_{\text{vector_noise}})$$

• $p_{\text{overlap}}$: Share of spans touching a bounding-box overlap above 0.25 (measured by span involvement so the ratio does not dilute on dense pages).
• $p_{\text{tab_entropy}}$: Set to 1.0 when $\ge 4$ consecutive $\ge 3$-island rows fail column bucketing (jagged column counts or $x_0$ standard deviation above tolerance).
• $p_{\text{vector_noise}}$: Density of diagonal vector segments intersecting text spans.
• Routing threshold: Regions with $S(R) \ge 0.45$, scanned pages without text spans, and diagram clusters ($\ge 4$ vector segments) are padded, cropped to WebP, and sent to the configured VLM endpoint.
• Font-anomaly floors (v0.2.0): regions additionally force $S(R) \ge 0.75$ when span mojibake (U+FFFD/PUA/control fallout from corrupt CMaps) exceeds 8%, and $S(R) \ge 0.55$ for math-font/symbol clusters and sub/superscript baseline oscillations — routed as CorruptedCMapOrMojibake / MathGlyphCluster.
• Deterministic math pre-pass (v0.3.0): lines reconstructible to LaTeX at confidence $\ge 0.80$ (math glyphs, _{}/^{} baseline shifts, \frac{}{} bars) emit $...$/$$...$$ directly and their MathGlyphCluster regions skip the VLM entirely.

Project Structure

razorpdf/
├── crates/
│   ├── razorpdf-core/            # Engine: extract -> geometry -> triage -> VLM -> emit
│   │   ├── benches/throughput.rs # Criterion pages/sec benchmark suite
│   │   └── examples/dump_fixtures.rs
│   ├── razorpdf/             # Binary: parse / inspect / bench / schema / diff / jsonl / mcp
│   └── razorpdf-py/              # PyO3 bindings: parse_file / to_markdown / search_chunks / render_citation / extract_schema / diff_pdfs / to_columnar_dict / to_jsonl (+ GIL release, .pyi stubs)
├── scripts/
│   ├── bench_wild.py             # Real-world PDF benchmark vs pypdf, pdfminer.six, pymupdf
│   └── bench_omnidoc.py          # OmniDocBench-style harness: throughput, RSS, NED, table TEDS
└── tests/wild/                   # Wild-PDF corpus (gitignored, refetchable — see README.md)

Roadmap

• Completed (Phases 1–6): Core geometric extraction, recursive XY-cut layout engine, R-tree vector table reconstruction, $S(R)$ triage scorer, sub-page WebP VLM client, MCP stdio server, and PyO3 bindings.
• Completed (v0.2.0): Corrupt-CMap/math-glyph triage routing, hyperlink + AcroForm extraction, cross-page table/paragraph stitching, header-preserving token chunking with page_spans provenance, visual citation overlays, blake3 VLM response cache, 7-tool progressive-disclosure MCP server (TOC/BM25/page-range/region-render), expanded Python bindings, and the OmniDocBench-style benchmark harness.
• Completed (v0.3.0): Deterministic spatial JSON Schema extractor (AcroForm/inline/spatial/table tiers with bbox provenance), running header/footer stripper + footnote graph linker ([^n]), deterministic math-to-LaTeX reconstructor (_{}, ^{}, \frac{}{}), columnar Arrow/Polars export + JSONL, structural PDF diff engine, 10-tool MCP server, schema/diff/jsonl CLI subcommands, 8-function Python API with PEP 561 stubs, and LaTeX-aware math triage.
• Planned:
  • pdfium render backend (--features pdfium)
  • Exact token counting via vendored tiktoken ranks (currently opt-in download + heuristic fallback)
  • Local ONNX micro-tier for offline formula/table recognition

Contributing

Contributions, bug reports, and benchmark additions are welcome! Feel free to open an issue or submit a pull request on GitHub.

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE or http://opensource.org/licenses/MIT)

at your option.