Skip to main content

document2md

test

Converts a PDF, or a set of scanned page images, into Markdown — any document, such as an edition of Mexico's official gazette (DOF, Diario Oficial de la Federación) — optionally cropped down to a single note. It has two backends: mineru (OCR and layout analysis, for scanned pages) and pymupdf4llm (reading a born-digital PDF's own embedded text layer, no OCR needed); document2md's own contribution is:

  • Keeping mineru's mineru-api server warm across a batch of documents, instead of paying its startup (and model-loading) cost once per document.
  • Stitching the OCR of a list of page images (several scanned pages of the same note) into one continuous Markdown document.
  • Rewriting the raw HTML tables mineru falls back to (rowspan/colspan) into Markdown tables, so the output is Markdown all the way through.
  • Cropping the result down to a single note, by locating its title and the next note's title in the OCR'd text — useful because a scanned page usually holds the tail of one note and the head of the next.

It was extracted from the LegalIA monorepo at commit e1f258c, where every commit of its earlier history (as packages/document2md, and as packages/dof2md before the rename) can still be read.

Documentation

Install

pip install document2md

converts born-digital PDFs (--backend pymupdf, via pymupdf4llm) out of the box — no OCR, no models. Note: PyMuPDF and pymupdf4llm are AGPL-3.0 licensed, unlike the rest of document2md (Apache-2.0); check that fits your project before redistributing. Scanned documents still need mineru's OCR/layout models — add the mineru backend for that:

pip install "document2md[mineru]"

--backend/BatchConverter(backend=...) name each backend explicitly so auto's policy (prefer mineru when installed, fall back to pymupdf otherwise) is visible rather than implicit. nota2md's ocr extra (in the LegalIA repository) must depend on document2md[mineru]>=0.4.0, not a bare document2md>=0.3.0, or pip install nota2md[ocr] stops installing mineru.

For development, from a clone of this repository:

pip install -e ".[mineru,test]"

Usage

CLI

document2md takes exactly one input source — a local PDF or a set of local page images — and converts it to Markdown. It never downloads anything itself; get the PDF first (e.g. dofjson.download_edicion_pdf for a whole DOF edition by date and edition, see the dofjson README), then convert it:

document2md --pdf edicion.pdf   # a local PDF

document2md --images pagina-1.jpg pagina-2.jpg \
    --filename out.md      # scanned pages, in order

--filename sets the output Markdown's name; with --pdf it defaults to the PDF's own name (edicion.pdf → edicion.md), but with --images it's required, since a set of images has no single name to derive one from. --outdir sets the output directory (default: output/).

Since one edition's PDF holds every note published that day, --titulo/--titulo-siguiente crop the resulting Markdown down to just one note — its own title, and the next note's title, as they appear in the gazette's own index:

document2md --pdf edicion.pdf \
    --titulo "ACUERDO por el que se..." \
    --titulo-siguiente "DECRETO por el que se..."

Title matching is fuzzy (OCR text rarely matches an index title exactly), so a match below --min-confidence (default 0.6) is treated as not found and the crop falls back to keeping more text rather than dropping content. Other flags:

  • --keep-pages — also keep the uncropped Markdown, as <outdir>/<pdf stem>.full.md.
  • --keep-mineru-output — keep mineru's own raw output (layout/model JSON, rendered PDFs...) in <outdir>/<pdf stem>_mineru/ instead of discarding it; useful when a conversion looks wrong and mineru's own read of the page is the first thing worth inspecting.
  • --backend {auto,mineru,pymupdf} (default auto) — which backend converts the document. auto resolves to mineru when it's on PATH, otherwise pymupdf. pymupdf reads a PDF's own embedded text layer instead of running OCR — far faster, at the cost of slightly worse structure — and can't handle scanned page images or a PDF without enough of a text layer; document2md exits with a message telling you to pip install "document2md[mineru]" instead of a traceback when that happens, or when mineru is requested but isn't installed.

Python: batch conversion

Converting many documents in one run is where mineru's startup cost starts to matter. BatchConverter keeps a single mineru-api server warm across the whole batch instead of restarting it per document:

from document2md import BatchConverter

jobs = [
    ("a.pdf", "output", "a.md"),
    (["b-p1.jpg", "b-p2.jpg"], "output", "b.md"),
]

with BatchConverter() as convert:
    for path_or_paths, outdir, filename in jobs:
        convert(path_or_paths, outdir, filename)

Each call takes a single PDF path, or a list of image paths for a document spanning several scanned pages, and writes the result to outdir/filename. The same titulo/titulo_siguiente, min_confidence, keep_pages and keep_mineru_output options the CLI exposes are also its keyword arguments — see BatchConverter.__call__'s docstring for the full signature. BatchConverter(backend="auto") (the default) picks which backend does the conversion: "mineru" when it's on PATH, otherwise "pymupdf"; the resolved name is available as convert.backend once entered. "pymupdf" raises RuntimeError (naming the mineru extra) rather than silently falling back to it on input it can't handle — a list of page images, or a PDF without enough of an embedded text layer.

nota2md.legal_provisions accepts an already-__enter__'d BatchConverter as its own converter parameter, so a batch of DOF legal provisions can share the same warm server too.

Tests

pytest tests
pytest dof2md/tests

Metadata

Release files for document2md 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for document2md 0.4.0
File Size Uploaded Provenance Actions
document2md-0.4.0.tar.gz 35.0 kB Without provenance Details for document2md-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for document2md 0.4.0
File Interpreter ABI Platform Actions
document2md-0.4.0-py3-none-any.whl Python 3 none any Details for document2md-0.4.0-py3-none-any.whl

Total release size: 60.6 kB

Release files / document2md-0.4.0.tar.gz

Download URL document2md-0.4.0.tar.gz
Size 35.0 kB
Tags Source
SHA-256 checksum
How to use checksums
120783f7a43c792d1278d219da442b112168fe3b7ad6129f6f977cc129dfd02d
BLAKE2b-256 checksum
How to use checksums
0e043f885b93c11105159f7fa666898983dfa17db355728b2a0ab00dd43872d5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Release files / document2md-0.4.0-py3-none-any.whl

Download URL document2md-0.4.0-py3-none-any.whl
Size 25.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2669dcc24f2a8c57761e4f2d5a3e440ab078bfaacd888d5a46626191372f4fb8
BLAKE2b-256 checksum
How to use checksums
b0e4a3893bbd34187877336027dc750605e4d8e28798ec8c4e18e49300b2bf15
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Security for document2md 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page