pdf-to-carve

Convert born-digital PDFs, scanned PDFs, and document images into validated Carve (.crv) source.

The conversion does not pass through Markdown and does not ask an AI model to write Carve syntax. Extraction produces a small, validated JSON document; a deterministic writer produces Carve.

See Why Carve over Markdown? and the equal-model complex-PDF benchmark. The conversion examples provide small before-and-after PDFs, validated JSON, Carve, and Markdown for direct inspection.

For the broader extractor comparison, see the reproducible competitor benchmark, including raw normalized outputs, pinned versions, completion failures, scoring code, and methodological limitations.

The stratified deterministic gold set publishes exact-document accuracy plus character and word error rates by document class.

Carve compatibility

Generated .crv files target the Carve 0.1 specification. Carve engine and binding releases use independent patch versions, so their package versions are not the specification version. For example, this repository's semantic round-trip tests currently use the published Python binding carve-lang==0.1.2, while the Carve specification remains 0.1.

The writer is also checked against current Carve development behavior and keeps a compatibility fallback where the published binding differs. See Carve's versioning policy and version map for the relationship between specification, engine, and binding releases.

PDF/image -> text, vision, or hybrid extraction -> document JSON -> Carve writer
                                                               -> carve fmt/lint
                                                               -> review report

Install

Python 3.10 or newer is required.

pip install pdf-to-carve

For development:

git clone https://github.com/markup-carve/pdf-to-carve.git
cd pdf-to-carve
uv sync --extra dev

Use

Auto mode uses deterministic PDF text extraction when a page has enough usable text, otherwise it selects vision:

pdf-to-carve document.pdf -o document.crv

Force a mode or select pages:

pdf-to-carve document.pdf --mode text --start-page 2 --end-page 8
OPENAI_API_KEY=... pdf-to-carve scan.pdf --mode vision --model gpt-4o-mini
OPENAI_API_KEY=... pdf-to-carve complex.pdf --mode hybrid --cache-dir .cache/pdf-to-carve
pdf-to-carve complex.pdf --mode hybrid --provider codex-cli --model gpt-5.6-sol
pdf-to-carve complex.pdf --mode hybrid --provider claude-cli --model sonnet
pdf-to-carve page.png --mode vision -o page.crv

Hybrid mode sends both rendered pages and positioned text evidence. It is useful for complex born-digital PDFs where text extraction preserves spelling but loses layout. Vision requests are capped at 20 pages by default; use --max-pages and --dpi to tune the request explicitly.

--provider codex-cli uses an existing authenticated Codex CLI installation when API billing is unavailable. It launches an ephemeral, read-only session, ignores repository rules and user configuration, validates the returned JSON locally, and otherwise uses the same hybrid pipeline. This is still a remote AI request—not local inference—and requires an image-capable Codex model.

--provider claude-cli similarly uses an existing authenticated Claude Code installation. It defaults to sonnet, disables project customizations and session persistence, exposes only Claude's Read tool, and grants that tool access only to isolated copies of the rendered pages. Any permission denial, CLI error, non-JSON response, or locally invalid document fails the conversion. It is also a remote AI request and is intended for explicit local use.

Use --base-url with an OpenAI-compatible Chat Completions endpoint. The configured model must accept image inputs and JSON-object response format. Transient network, rate-limit, and server failures are retried three times by default; adjust this with --retries.

Save the provider-neutral extraction result and replay it without an API call:

pdf-to-carve scan.pdf --mode vision --save-json scan.crv.json -o scan.crv
pdf-to-carve scan.crv.json --from-json -o rebuilt.crv

Surface confidence beside each generated block and run the local validated correction loop over uncertain or warned blocks:

pdf-to-carve scan.crv.json --from-json --correct \
  --save-json corrected.json --annotate-confidence -o corrected.crv

The annotations are Carve comments and contain block number, page, confidence, and warning count - never evidence text. Blocks without provenance are marked with unknown page and confidence. The correction loop accepts only extraction-model JSON that passes the same strict validation as a provider response. See the review workflow.

Generate a self-contained local review report and extract embedded raster assets:

pdf-to-carve report.pdf --mode hybrid -o report.crv \
  --review-html review.html --assets-dir assets --save-json report.crv.json

The report escapes all document content and includes the generated source, validated JSON, provenance coverage, and warning count. See the review workflow.

For the strongest check, point to the official Carve CLI. The command returns exit status 2 if carve fmt --check or carve lint reports a problem:

pdf-to-carve document.pdf -o document.crv --carve-command carve

When AI is used

AI is deliberately narrow and optional:

• Text PDFs use PDFium by default and make no network request.
• Scans, images, and complex layouts use one document-level vision request.
• Hybrid mode gives the model bounded positioned-text evidence alongside images.
• The model returns JSON, never executable content or final Carve syntax.
• JSON can be inspected, versioned, corrected, and replayed offline.
• No agents, embeddings, vector store, or automatic rewriting are involved.

Auto mode currently uses text coverage as its conservative routing signal. Complex born-digital tables can still benefit from explicitly selecting --mode vision.

Extraction model

The version-1 model accepts headings, paragraphs, flat lists, code blocks, quotes, admonitions, spanning tables, figures, thematic breaks, and page breaks. Inline content can represent text, common decoration, super/subscript, critic markup, inline footnotes, code, math, and links. Optional provenance records associate a block with a page, bounding box, confidence, warnings, and short evidence for review without affecting Carve output. Unknown fields and malformed nodes fail closed with a precise JSON path. The machine-readable contract is document-v1.schema.json.

This is intentionally smaller than the complete Carve AST. It describes what document extraction can establish reliably, while the writer owns syntax and escaping. The contract can evolve by adding a new version.

Limitations

• Deterministic text mode uses document-relative font sizes and positioned text to recover headings, wrapped paragraphs, conservative lists and simple tables, monospaced code blocks, bold/italic runs, super/subscript, external link annotations, geometrically supported underline/highlight, and conservative indented quotations. It conservatively infers strongly evidenced code languages and table alignment. With --assets-dir, it also preserves obvious multi-part vector figures as cropped PNGs with visible-label alt text and places embedded raster figures. Repeated headers, footers and page numbers, strong two- and three-column layouts, explicit captions, internal page links, rotated text, multiline numeric superscript footnotes, reused raster assets, and repeated multi-page table continuations are handled conservatively. It does not invent code indentation, reconstruct arbitrary diagram semantics, or guess ambiguous unbordered table spans. Bordered spans are recovered only when enclosing cell geometry proves their row and column extent.
• Vision accuracy and cost depend on the selected provider and model.
• Hybrid mode uses deterministic text extraction as its trusted baseline. Model wording changes are discarded; code may change whitespace only. Structural diagrams require high-confidence positional provenance, while deterministic table alignment and extracted asset references take precedence. Accepted repairs retain remapped provenance, and rejected repairs appear in diagnostics.
• --assets-dir extracts and deduplicates embedded raster images. The PDFium text path additionally crops clear multi-part vector figures; ambiguous figure regions and exact provider-placeholder matching remain manual review steps.
• Very large PDFs may exceed a provider's request limits. Select a page range.
• PDF content is untrusted input. Review converted documents before publishing.

See privacy and security, including the cloud-vision redaction checklist, before processing sensitive files. For distribution planning, see the PDF backend licensing options.

The default PDFium backend is permissively licensed. Install the optional PyMuPDF compatibility backend when exact legacy behavior is required:

pip install 'pdf-to-carve[pymupdf]'
pdf-to-carve document.pdf --pdf-backend pymupdf

Development

uv run ruff check .
uv run ruff format --check .
uv run pytest --cov=pdf_to_carve --cov-report=term-missing
uv run python benchmarks/gold/score.py --output benchmarks/gold/results.json \
  --report benchmarks/gold/REPORT.md
uv build