mag-pdf

PDF extraction with two profiles and one result shape.

	fast (default)	tables
What	LiteParse behind an OCR gate	DocLayout-YOLO + TableFormer over the pypdfium2 text layer
For	most documents; scanned pages via Tesseract	documents whose value is in their tables
Install	pip install mag-pdf (one dependency)	pip install 'mag-pdf[tables]'
Cost	~100-200 ms with no OCR needed	~0 for a plain page; a model only where the gate says so

from magpdf import extract, warmup

warmup()                       # optional, ~160 ms, at worker startup
doc = extract(pdf_bytes)

doc.markdown                   # whole document, one unified string
doc.pages                      # per-page, 1-indexed page_number
doc.ocr_pages                  # which pages OCR actually ran on
doc.timings                    # {"gate_ms", "text_ms", "ocr_ms", "total_ms"}

Both profiles return the same Document, so a caller can switch with one keyword and read the result the same way.

Why

OCR costs ~0.2-1.3 s per page and has no warm-up to amortise (a reused parser measured 5,797 ms against a fresh one's 5,864 ms). So the only lever on latency is not OCRing a page at all.

magpdf reads LiteParse's own per-page signals in 6-105 ms and OCRs a page only when it can add something the text layer cannot:

Trigger	Fires when	Why it earns its cost
image	page carries substantial raster content	recovers tables and text rendered inside images. On one corpus page a table went from 0 markdown pipes to 16
dead_text	text layer garbled or near-empty	catches true scans, and fonts subset without ToUnicode CMaps, where a text layer exists but decodes to bullet glyphs. One corpus file: 160 chars to 7,315
table (off)	table detected in the text layer	measured across 53 such pages: zero change to table structure for ~72 s of OCR

Over a 232-page corpus that OCRs 55% of pages instead of 100%, with identical output.

Latency

A 10-page PDF, warm. OCR page count is the only variable that matters:

Pages needing OCR	Total
0	~100-200 ms
~5	~5-6 s
10	~13 s

The gate itself is under 1% of any run that OCRs.

Running one document at a time? Set num_workers=4. The default of 1 is tuned for concurrent Temporal activities and is 2.8x slower on OCR-heavy documents (14,779 ms vs 5,318 ms on a 4-page all-OCR file). See docs/KNOWLEDGE_HANDOFF.md section 5.

Configuration

from magpdf import ExtractConfig, OcrTriggers, extract

doc = extract(pdf_bytes, config=ExtractConfig(
    triggers=OcrTriggers(image=True, dead_text=True, table=False),
    dead_text_coverage=0.02,
    max_ocr_pages=100,      # cap; dead_text pages win, rest reported as skipped
    num_workers=1,
    ocr_enabled=True,       # False = text layer only
))

There are no document-class heuristics in the gate. It reads general per-page signals and applies them uniformly to every PDF. Retuning is config, never a code change.

Vision escalation (optional)

A page that carried an image and still came back near-empty is where Tesseract most plausibly failed. Inject a client and those pages get a second look:

doc = extract(pdf_bytes, vision_client=my_client)
doc.escalated_pages          # [3]

The client only needs ocr_image(image_bytes, media_type, *, prompt=None, max_tokens=...) -> (text, usage). That is a structural match for mag-file-handler's VisionClient, so an existing client works as-is - but neither package imports the other.

Without a client, magpdf is fully offline, deterministic, and costs no tokens.

---

The tables profile

pip install 'mag-pdf[tables]'

from magpdf import extract, warmup

warmup(profile="tables")               # loads both models; seconds, not ms
doc = extract(pdf_bytes, profile="tables")

doc.markdown                           # one unified, ordered document
doc.table_pages                        # pages TableFormer actually ran on
doc.text_pages                         # pages served free from the text layer
doc.warnings                           # non-fatal degradations, in the result

Per page, once: render, ask a layout model what is on the page, and pay for TableFormer only where it is worth it. A plain digital page is served from the pdfium text layer for zero model time.

page ─ render once @2.0 ─┬─ YOLO ─→ table? figure? sparse text?
                         │              │
                         │        ┌─────┴──────┬────────────┐
                         │      table       figure       neither
                         │        │            │            │
                         │   TableFormer   vision (if a   pdfium
                         │                  client)       text layer
                         └─ text cells ────→ tokens for cell matching

It does not use docling. docling_ibm_models never imports it, so the two models are called directly. That avoids monkey-patching docling's private layout-engine factory, which is what the pipeline this ports has to do, and it means no DocumentConverter, no temp file, and no pinned orchestration layer.

Output

Clean GitHub-flavoured markdown. Tables become pipe tables, titles become # headings, and blocks are emitted in reading order so a caption stays with its table. Pages are joined in order into one document — Document.markdown is the artifact, not a starting point.

GPU

from magpdf import ExtractConfig, TablesConfig, extract

cfg = ExtractConfig(profile="tables", tables=TablesConfig(device="auto"))
doc = extract(pdf_bytes, config=cfg)

table_workers defaults to 0, meaning derive it from the device: a page's tables are predicted concurrently on an accelerator and sequentially on CPU. That is not a hedge — TableFormer is already parallel on CPU via num_threads, so N concurrent tables there oversubscribe the same cores and every table gets slower. Set table_workers explicitly to override either way.

Vision (optional)

Inject a VisionClient and two things become possible: a scanned page is replaced by its transcription, and a figure description is inserted at the figure's position. With no client injected, neither runs and the profile stays fully offline and deterministic.

Concurrency

extract(profile="tables") serialises at process scope. Concurrent callers block; they never race and never crash. Scale with processes, not threads.

pypdfium2 is not thread-safe. This is enforced rather than documented, so a threaded caller gets slow instead of corrupt.

Known limits

• Reading order is a (top, left) sort. Correct for single-column documents; it interleaves columns line by line on multi-column layouts.
• The extra is heavy. docling-ibm-models pulls torch, torchvision, transformers and accelerate — on the order of a couple of GB.
• Weights are fetched once from HuggingFace and cached. With no access, the profile degrades to the pdfium text layer and says so in Document.warnings; it does not hang or fail.

---

Scope

PDF only. LiteParse parses PDF natively and reaches every other format by shelling out to LibreOffice (measured: ~9.5 s for one XLSX, returned as 36 paginated "pages"). Office, email, HTML and tabular formats belong to mag-file-handler, which this package neither imports nor is imported by.

Non-PDF input raises UnsupportedFormat. Encrypted PDFs return ok=False, error="pdf is encrypted" unless a password= is supplied.

Known ceilings

LiteParse reconstructs tables by spatial column projection and does not detect where a table ends, so it can absorb adjacent content into the last row - identically with OCR on or off. Tesseract also drops cells on image-tables. Both are documented with evidence in docs/KNOWLEDGE_HANDOFF.md section 7. This package wraps those flaws and reports them; it does not claim to fix them.

Command line

Installing the package puts two commands on PATH, one per pipeline:

magpdf-fast   in.pdf -o out.md      # gated LiteParse
magpdf-tables in.pdf -o out.md      # DocLayout-YOLO + TableFormer

Two commands rather than one with a --profile flag: they do not share dependencies or failure modes, and one typo should not silently run the other engine. Markdown goes to stdout, every diagnostic to stderr, so > out.md yields a clean file.

magpdf-fast  - < in.pdf > out.md       # stdin/stdout
magpdf-tables a.pdf b.pdf --out-dir o  # batch; models load once, not per file
magpdf-tables in.pdf --json            # metadata envelope (JSON Lines if batched)
magpdf-fast  --self-test               # does OCR actually work on this platform?
magpdf-tables --doctor                 # environment + where models resolve from

Exit codes: 0 ok, 1 extraction error, 2 usage, 3 not a PDF, 4 engine or models unavailable.

--workers defaults to 4 here, not the library's 1. The library default is tuned for a caller that already runs documents in parallel; a CLI is the opposite case, and 4 measured ~2.8x faster.

magpdf-tables warms the models up before extracting. This is deliberate: a missing model does not make extract() raise - the layout gate swallows it and the document degrades to plain text-layer markdown with no warning and exit 0. The pre-flight turns that into exit 4. --no-warmup opts out.

Serve (sidecar)

Both pipelines over HTTP, for callers that are not Python or not on the same box:

pip install 'mag-pdf[tables,serve]'
magpdf-serve --port 8109

GET  /healthz   {"status","fast_ready","tables_ready",...}
POST /extract   multipart: file, profile=fast|tables, password?, table_mode?

curl -sf -F file=@in.pdf -F profile=tables localhost:8109/extract | jq -r .markdown

The response body is the same envelope magpdf-fast --json prints. One shape, so a caller can move between the CLI and the sidecar without a second parser.

• /healthz returns 503 until both profiles are warm, and stays 503 if the tables models never load. That is deliberate: a missing model does not make extract() fail, it makes it quietly return text-layer markdown, so readiness is the only place that failure can be made visible.
• A bad document is 200 with ok:false (an encrypted PDF is a fact about the file, not an outage); a bad server is 4xx/5xx. Callers need to tell those apart.
• Run one worker per container. The tables pipeline serialises on a process-scope lock, and a second worker in the same container loads its own copy of the models for no extra throughput. Scale with replicas.

Environment: MAGPDF_SERVE_TABLES, MAGPDF_FAST_CONCURRENCY, MAGPDF_FAST_WORKERS, MAGPDF_TABLE_MODE, MAGPDF_NUM_THREADS, MAGPDF_MAX_UPLOAD_BYTES, MAGPDF_QUEUE_TIMEOUT_SECONDS, LOG_LEVEL.

Docker

One image, all three entry points, weights baked in. It never touches the network.

docker pull veermedi/mag-pdf:0.3.0

docker run --rm -v "$PWD:/data" veermedi/mag-pdf:0.3.0 \
    magpdf-tables /data/in.pdf -o /data/out.md

docker run --rm -i veermedi/mag-pdf:0.3.0 magpdf-fast - < in.pdf > out.md
docker run --rm --network none veermedi/mag-pdf:0.3.0 magpdf-tables --self-test

docker run --rm -p 8109:8109 veermedi/mag-pdf:0.3.0 magpdf-serve

• Never pass -t when capturing output. A TTY merges stderr into stdout and translates newlines to CRLF, corrupting the markdown.
• Writing to a bind mount as a non-root container user needs --user "$(id -u):$(id -g)" on Linux, or the write is denied.
• The image is built linux/amd64 only. This is now a build choice rather than a constraint: liteparse 2.13.0 does publish a Linux arm64 wheel, so an arm64 image is buildable if someone wants one. Until it is built, Apple Silicon needs --platform linux/amd64 and should expect emulation to be slow.
• The tables profile serialises to one document per process, so scale with replicas or docker run invocations, never threads.

Build it yourself:

docker build -t mag-pdf:local ./magpdf

The build runs six gates and fails rather than shipping something subtly wrong: CPU-only torch, models loading from the baked cache, a real tables extraction with no degradation warnings, the OCR canary, an assertion that only the TableFormer weights are baked in and the docling orchestration package is absent, and that the HTTP surface builds and still emits tables a downstream chunker can find.

Install

pip install mag-pdf

Python 3.10+, except on macOS, which needs 3.11+: liteparse publishes no macOS cp310 wheel, and pip's fallback of building the Rust sdist omits the bundled pdfium library — so the install succeeds and every parse then dies with PanicException: failed to load pdfium shared library. Linux and Windows are fine on 3.10. CI excludes that one cell rather than pretending it works.

Apache-2.0.