⚡ LightningParse

Fast, accurate PDF parsing for RAG pipelines — a Rust extraction core (via PyO3) returning structured JSON, with automatic OCR fallback for scanned pages.

Status: published on PyPI — pip install lightningparse. Prebuilt wheels for CPython 3.8–3.14 on Linux (x86_64, aarch64), Windows (x64), and macOS (x86_64, arm64). Core pipeline complete and benchmarked end-to-end.

Install

pip install lightningparse

Tier 1 (digital-native) extraction works out of the box with no system dependencies.

OCR (Tier 2) additionally requires Tesseract on your PATH. It is invoked only when a page has no extractable text layer. Without it, scanned pages raise OcrMissingDependencyError rather than failing silently.

git clone https://github.com/ShivamMalge/LightningParse.git
cd LightningParse/lightningparse-core
maturin develop --release   # requires a Rust toolchain

Quickstart

parse_pdf() returns a JSON string, so parse it before use:

import json
from lightningparse import parse_pdf

result = json.loads(parse_pdf("document.pdf"))

for page in result["pages"]:
    for block in page["blocks"]:
        role = block.get("block_role")  # "heading", "code", or None
        print(block["section_id"], role, block["text"][:80])

print(result["metadata"]["tier"])  # "digital", "scanned", or "mixed"

# Pages using a content-stream filter that can't be decoded are reported here
# rather than silently degrading to OCR.
if result["metadata"].get("warnings"):
    print("Warnings:", result["metadata"]["warnings"])

Fatal parse errors raise CorruptPdfError instead of returning an empty result, so failures surface in your pipeline rather than passing as empty documents:

from lightningparse import parse_pdf, CorruptPdfError

try:
    result = json.loads(parse_pdf("maybe_broken.pdf"))
except CorruptPdfError as e:
    print("Unparseable:", e)

What's New in v0.5.0

• Fixed silent content loss in header/footer detection. Margin bands were a fraction of content extent rather than the page's real height, so the band reached below the true margin into body text — and blocks tagged as page furniture are dropped before chunking. Across 7 documents, 19 blocks of real content were being deleted, including document titles, author lines and chapter headings. Bands now derive from real page geometry.
• Removed the page-1-only header/footer fallback, which tagged top/bottom blocks on position alone with no cross-page corroboration and caused 8 of those 19 deletions. Its footnote branch is retained.
• Page geometry exposed in the schema: optional page_width / page_height per page, from /CropBox or /MediaBox, inherited via a cycle-safe /Parent walk, axes swapped for /Rotate 90|270. Falls back to previous behaviour when absent.

Behaviour change: fewer blocks are tagged as page furniture, so more content reaches downstream consumers. Strictly one-directional — 19 blocks freed, 0 newly tagged.

Why

Traditional Python PDF libraries (PyPDF2, pdfplumber, PyMuPDF) are GIL-bound and process pages sequentially, which becomes a bottleneck in RAG ingestion pipelines. LightningParse pushes extraction, header/footer cleanup, and OCR fallback into Rust, parallelized across pages, and returns structured JSON that Python can chunk with page/section metadata intact.

Architecture

Two processing tiers, selected per page:

• Tier 1 — Digital-native PDFs: direct text extraction, no OCR. This is where the speed claim is benchmarked.
• Tier 2 — Scanned/image PDFs: routed per-page to OCR (Tesseract) when no text layer is present.

Documents containing both report tier: "mixed".

Full design details: ARCHITECTURE.md · Product scope: PRD.md

Benchmarks

LightningParse is 16.3×–53.9× faster than pypdf and 44.8×–84.4× faster than pdfplumber on the representative digital-native (Tier 1) documents below, with the gap widening on longer documents:

Absolute milliseconds are machine-dependent and will vary with hardware, thermal state and background load. The portable claim is the speedup ratio — the baselines are timed on the same machine in the same run, so hardware cancels out. This session's own investigation demonstrated it: these absolute figures moved ~2x from the previous published set while pypdf and pdfplumber (code LightningParse does not touch) moved 2.4-3.5x, i.e. the shift was hardware, not the codebase. Figures are the median of 25 timed runs after 10 warm-up runs — the warm-up matters, because LightningParse's first runs measure markedly slower than its steady state. See methodology.

Trivial single-page synthetic fixtures span wider in both directions (3.8×–427.4×), because at that size the ratio is dominated by fixed overhead rather than extraction work. They are not quoted as headline figures.

OCR (Tier 2) and mixed-document handling are benchmarked separately — pypdf and pdfplumber can't perform OCR, so comparing their near-instant-but-empty results against actual OCR time would mislead rather than inform.

A concurrent-load test confirms the Rust FFI genuinely releases Python's GIL during parsing: 10 concurrent OCR-heavy parse requests complete 4.78× faster than running them sequentially, on an 8-core/16-thread machine.

Full methodology and per-document results: benchmarks/BENCHMARKS.md

Known Limitations

• Page furniture is under-removed on long documents: header/footer detection requires a repeated block to appear on ≥70% of pages. Running heads usually change per chapter, so on a long book nothing clusters that widely and no furniture is removed at all — measured on a 1475-page textbook whose site-wide footer appears on 734 pages and is still not stripped. Separately, a bare page number normalises to an empty string during clustering and can never be tagged. Both mean folios and running heads can flow into body text and into chunks.
• Corrupt ASCII85 streams fail silently to OCR: a corrupt or truncated ASCII85Decode stream is not reported. The underlying decoder returns the raw undecoded bytes on failure rather than an error, so the page yields no text and falls through to OCR, reported as tier: "scanned" with an empty warnings array — indistinguishable from a genuine scan. Undecodable filters are still reported normally; this affects only corrupt data within a supported filter. Tracked for a future release.
• Content stream filters outside the supported set: FlateDecode, LZWDecode, ASCII85Decode, ASCIIHexDecode, and RunLengthDecode are decoded. PDFs using any other filter (e.g. JBIG2Decode, or a /Crypt filter) produce no Tier 1 text and are routed to OCR, with a warnings entry so callers can detect it.
• Monospace detection on composite fonts: the structural uniform-width check applies only to simple fonts with a /Widths array. CID/Type0 fonts bypass it and fall back to font-name matching ("courier", "mono"), so an unconventionally-named monospace CID font may not be tagged block_role: "code". Glyph widths for CID fonts are parsed correctly from /W and /DW.
• Heading detection false positives: classification uses font-size ratio, weight, and line length relative to the document's own body text, with no semantic understanding. Stylistically-emphasized text that isn't a real heading (a bolded date range, a pull-quote) can be misclassified.
• Complex or borderless tables: table detection requires a nearby caption (e.g. "Table 1") and consistent row/column geometry. Tables without captions, or with irregular formatting, fall back to flat text — no data is lost, but structure isn't always recovered.
• OCR quality on degraded scans: Tesseract confidence filtering removes most scan artifacts, but heavy combined distortion (rotation + noise + lighting gradient + blur) can cause the filter to discard real content along with the noise. The system fails safely — no crash, no hallucinated text — but does not recover text from severely degraded scans.
• Encrypted and form PDFs: not explicitly supported.

Scope

In scope: digital-native PDF extraction, header/footer/footnote removal, semantic block typing, OCR fallback for scanned pages, and metadata-aware output suitable for chunking.

Not in scope yet: encrypted/form PDFs, ML-based layout detection. See PRD.md §2 — these are deliberate cuts, not oversights.

Contributing

See AGENTS.md for repo conventions, build commands, and non-negotiable rules (FFI safety, GIL handling, benchmark discipline) before opening a PR.

License

MIT License.