PDF to Markdown

Convert PDFs to clean Markdown and RAG-ready JSONL chunks, locally:

• Reading order for one- and multi-column pages (column gutters are found before rows, so two columns are read column by column), rotated pages, and sideways text on a normal page (kept as its own paragraph after the page text).
• Headings from font size and weight, nested by section numbers (3.2.1 sits under 3.2); lists (bullets and numbers); code blocks from monospaced text; links ([text](url)); footnotes ([^1] references with the notes at the end).
• Tables: ruled tables, tables ruled only between columns or every few rows, and rule-bounded tables without vertical lines (booktabs style). Plain tables become Markdown tables; merged cells become HTML tables with rowspan/colspan.
• Clean text: repeated headers/footers, page numbers and running heads removed (the first copy of a repeated slide title, table caption or form line stays; the summary lists removed examples; numbered sentences and bold chapter labels are kept); Markdown characters in the PDF text (*, _, `, <, [, ]) are backslash-escaped; line-end hyphenation repaired (com-pile becomes compile, English-to-German keeps its hyphen); ligatures (fi, fl, ...) expanded.
• Images optionally extracted as PNGs (deduplicated) and linked from the Markdown in a ZIP.
• Page ranges (1-5,9,12-).
• Chunks for retrieval: JSONL with id, source, text, tokens, pages, heading_path, start, end. Each chunk is an exact slice markdown[start:end]; chunks start at headings, never exceed the token limit (estimated conservatively, see Use), repeat about overlap tokens when a section is split, and together cover all non-whitespace text.

Runs offline. Scanned PDFs without a text layer are reported clearly ("no text layer"); there is no OCR.

Install (no administrator)

irm https://raw.githubusercontent.com/chayprabs/localtools/main/pdf-to-markdown/install.ps1 | iex

curl -fsSL https://raw.githubusercontent.com/chayprabs/localtools/main/pdf-to-markdown/install.sh | bash
# Or, with Python 3.12+ on any OS:
pipx install localtools-pdf-to-markdown

Requires Windows 10/11, macOS 13+, or Linux (glibc).

Portable ZIPs: Windows, macOS and Linux releases.

Use

Run localtools-pdf-to-markdown and the UI opens at http://127.0.0.1:47104. Choose a PDF, optionally set pages, chunk size and overlap, tick "Extract images", and Convert. Preview the rendered Markdown, the raw Markdown or the chunks table, then download the .md, the ZIP with images, or the .jsonl.

localtools-pdf-to-markdown convert paper.pdf -o paper.md
localtools-pdf-to-markdown convert paper.pdf -o paper.zip          # Markdown + images/page-N-K.png
localtools-pdf-to-markdown convert book.pdf --pages 23-30 > ch1.md
localtools-pdf-to-markdown chunks paper.pdf --max-tokens 512 --overlap 64 -o chunks.jsonl

Options: --pages, --images (with -o FILE.zip), --max-tokens 64-8192 (default 512), --overlap (0 to half of max, default 64). Exit code 0 on success, 2 on error (one line on stderr); warnings (e.g. pages without a text layer) go to stderr. Outputs are written to a temporary file and renamed, so a failure never leaves a partial file.

tokens is a conservative estimate without a tokenizer download: words, digits and symbols for Latin-script text, about 2.2 per Chinese/Japanese/Korean character and extra weight for other non-ASCII characters and garbled Latin text from fonts without a Unicode map. Measured against OpenAI's cl100k tokenizer on English papers, code, tables, Arabic, Cyrillic, Chinese and Japanese text, it was never below the real count on any sampled passage of 300+ characters (lowest ratio 1.04; typically 1.5x for English, 1.1-1.8x for CJK), so chunks stay under the limit for cl100k-style tokenizers. Other tokenizers may differ.

API: POST /api/convert (multipart: file; fields pages, max_tokens, overlap, images = 1/0, format). With format=markdown, jsonl or zip the response is that file. The default (json) returns the summary, the Markdown (first 2 million characters), the first 500 chunks and a job id; GET /api/file?job=<id>&name=document.md|chunks.jsonl|document.zip|images/page-N-K.png serves the full files of the last three jobs while the tool runs.

curl -F file=@paper.pdf -F format=markdown http://127.0.0.1:47104/api/convert

Limits

64 MiB and 1,000 converted pages per PDF (use a page range for longer documents), 200,000 characters and 100,000 drawing objects per page, 2,000 images (40 megapixels each, 400 in total), 20 million characters of Markdown, 120 seconds per conversion, two conversions at a time (more get HTTP 503). Each conversion runs in a separate worker process that is killed on timeout, when the browser cancels, or when the tool exits. Workers get a 2 GiB memory limit on Windows (job object) and Linux (address space); macOS does not allow that limit, so there the page and object caps bound memory. Temporary folders are deleted after each job; leftovers from a crash are swept at the next start.

Engine and why

PDFium (Chrome's PDF engine) through pypdfium2 5.14.0 (Apache-2.0/BSD-3-Clause) supplies characters with positions, font size and weight, link annotations, vector paths and images, and repairs broken PDFs. The layout analysis on top (XY-cut reading order, lines, paragraphs, headings, lists, footnotes, header/footer removal, chunking) is this tool's own code; ruled-table detection adapts pdfplumber's algorithm (MIT) onto PDFium's paths.

Alternatives considered: docling (MIT) has ML layout and table models, but pulls PyTorch and model weights (gigabytes, no macOS Intel builds for current PyTorch), far too heavy for a small offline tool. pdfplumber/pdfminer.six (MIT) were the first candidate; pdfminer's pure-Python parsing measured 0.3-1.7 s per page on these test PDFs against 0.02-0.04 s with PDFium, so PDFium does the parsing. PyMuPDF / pymupdf4llm are AGPL and are used only in the tests, as an independent reference.

Measured against PyMuPDF's text (share of its words we keep / reading-order similarity): ResNet (CVPR two-column) 0.993 / 0.910, TraceMonkey (ACM two-column) 0.997 / 0.969, Attention Is All You Need 0.989 / 0.937, IRS Form W-9 0.995 / 0.979, Think Python chapter 1 0.980 / 0.989. The 64 data rows of the IRS Pub 15-T weekly wage-bracket table match PyMuPDF's table cells exactly.

Not included

• Right-to-left scripts (Arabic, Hebrew) are read in logical order line by line; mixed left-to-right runs inside an RTL line (numbers, Latin words) may come out in the wrong order. Arabic presentation forms are expanded to their letters.
• OCR: scanned pages have no text layer and are reported as such.
• Equations: math is kept as the text the PDF contains (often imperfect), not LaTeX.
• Tables without any rules (aligned by whitespace only) stay as text lines; booktabs tables (horizontal rules only) are detected.
• Inline bold/italic are not marked (only headings use font weight).
• Password-protected PDFs: remove the password first.
• Text drawn inside figures (chart labels) is kept as paragraphs, and fonts without a Unicode map produce dropped characters (a warning names the pages).

Update / uninstall

localtools-pdf-to-markdown update / localtools-pdf-to-markdown uninstall. If you installed with pipx: pipx upgrade localtools-pdf-to-markdown / pipx uninstall localtools-pdf-to-markdown.

MIT license. Bundled components: see THIRD_PARTY_NOTICES.md.