dof2md

Converts a PDF or a set of scanned page images from Mexico's official gazette (DOF, Diario Oficial de la Federación) — or any other document — into Markdown, optionally cropped down to a single note. It's a wrapper around mineru for the OCR/layout analysis itself; dof2md's own contribution is:

• Keeping mineru's mineru-api server warm across a batch of documents, instead of paying its startup (and model-loading) cost once per document.
• Stitching the OCR of a list of page images (several scanned pages of the same note) into one continuous Markdown document.
• Rewriting the raw HTML tables mineru falls back to (rowspan/colspan) into Markdown tables, so the output is Markdown all the way through.
• Cropping the result down to a single note, by locating its title and the next note's title in the OCR'd text — useful because a scanned page usually holds the tail of one note and the head of the next.
• Downloading a DOF edition's PDF directly, by date and edition.

Part of the LegalIA monorepo.

Install

pip install -e ".[test]"

Usage

CLI

dof2md takes exactly one input source — a date, a local PDF, or a set of local page images — and converts it to Markdown, unless --download-only is given with a date, in which case it stops after the download.

Given a date, it downloads that edition's PDF from the DOF site and converts it:

dof2md 2010-01-05                     # morning edition (default)
dof2md 2010-01-05 --edition VES       # evening edition
dof2md 2010-01-05 --outdir my_folder  # output directory

--download-only stops after the download, without converting — useful when only the PDF itself is needed (e.g. to archive it, or convert it later with --pdf):

dof2md 2010-01-05 --download-only     # writes only 05012010-MAT.pdf

This writes <date>-<edition>.pdf and <date>-<edition>.md to the output directory. Since one edition's PDF holds every note published that day, --titulo/--titulo-siguiente crop the resulting Markdown down to just one note — its own title, and the next note's title, as they appear in the gazette's own index:

dof2md 2010-01-05 \
    --titulo "ACUERDO por el que se..." \
    --titulo-siguiente "DECRETO por el que se..."

Title matching is fuzzy (OCR text rarely matches an index title exactly), so a match below --min-confidence (default 0.6) is treated as not found and the crop falls back to keeping more text rather than dropping content. Other flags:

• --keep-pages — also keep the uncropped Markdown, as <outdir>/<pdf stem>.full.md.
• --keep-mineru-output — keep mineru's own raw output (layout/model JSON, rendered PDFs...) in <outdir>/<pdf stem>_mineru/ instead of discarding it; useful when a conversion looks wrong and mineru's own read of the page is the first thing worth inspecting.

A date always downloads from the DOF site; to convert a document you already have, use --pdf or --images instead — no download involved:

dof2md --pdf edicion.pdf   # a local PDF

dof2md --images pagina-1.jpg pagina-2.jpg \
    --filename out.md      # scanned pages, in order

--filename sets the output Markdown's name; with --pdf it defaults to the PDF's own name (edicion.pdf → edicion.md), but with --images it's required, since a set of images has no single name to derive one from. --titulo/--titulo-siguiente and the other flags above work the same way regardless of the input source.

Python: batch conversion

Converting many documents in one run is where mineru's startup cost starts to matter. BatchConverter keeps a single mineru-api server warm across the whole batch instead of restarting it per document:

from dof2md import BatchConverter

jobs = [
    ("a.pdf", "output", "a.md"),
    (["b-p1.jpg", "b-p2.jpg"], "output", "b.md"),
]

with BatchConverter() as convert:
    for path_or_paths, outdir, filename in jobs:
        convert(path_or_paths, outdir, filename)

Each call takes a single PDF path, or a list of image paths for a document spanning several scanned pages, and writes the result to outdir/filename. The same titulo/titulo_siguiente, min_confidence, keep_pages and keep_mineru_output options the CLI exposes are also its keyword arguments — see BatchConverter.__call__'s docstring for the full signature.

nota2md.legal_provisions accepts an already-__enter__'d BatchConverter as its own converter parameter, so a batch of DOF legal provisions can share the same warm server too.

Tests

pytest -v