brewdoc

Document to Markdown for agent systems: PDF, Word, PPTX presentations, spreadsheets, EPUB and static HTML in, one Markdown file plus a JSON receipt out. No models or OCR. Local conversion is offline and deterministic (same file, same bytes). Explicit --url acquires a page. The receipt says what was rendered, what was dropped and what the route structurally cannot carry.

Install

Version 0.3.0 is prepared for publication. These registry commands apply only after 0.3.0 is published on PyPI. Original 0.2.0 publication is tracked separately.

The existing original source tag can be installed now. It predates EPUB and URL acquisition:

uvx --from git+https://github.com/kochetkov-ma/brewdoc@v0.2.0 brewdoc file.pdf

Requires Python >= 3.12. Base runtime dependencies: pdfplumber (pulls pypdfium2, Pillow, cryptography) and python-calamine. HTML requires CPython 3.12-3.14 and the native Lexbor parser from this exact conditional dependency:

selectolax==0.4.13; python_version < '3.15' and platform_python_implementation == 'CPython'

Other runtimes retain the .html route and Python API but explicitly refuse HTML. On supported CPython, a missing or broken parser is an environment failure and --self-check exits 1. On an unsupported HTML runtime, self-check verifies HTML refusal and checks all other formats.

Version 0.3.0 includes URL acquisition and optional JavaScript capture. After publication, install the extra with pip install 'brewdoc[render-js]==0.3.0'. It uses QuickJS-ng 0.17.0.1 and bundled LinkeDOM 0.18.13. From this source checkout, install the locked extra with uv sync --locked --extra render-js. Run it with uv run --locked --extra render-js brewdoc --url URL --render-js. No Node, browser engine or invocation-time component download is used.

Usage

brewdoc file.pdf --out file.md
brewdoc slides.pptx --out slides.md
brewdoc book.epub --out book.md
brewdoc page.html --out page.md
brewdoc --url https://example.com/article --out article.md
brewdoc --url https://example.com/article --render-js --timeout 30 --out article.md
brewdoc file.xlsx                # no --out: receipt line, then the Markdown, on stdout
brewdoc file.xlsx --sheet Calc --sheet Inputs
brewdoc file.xlsm --artifact formula/sheet/000002=formulas.json
brewdoc file.xlsm --artifact vba/project/000001=vbaProject.bin
brewdoc --self-check             # renders synthetic fixtures twice, compares sha256, exit 0 when green

--sheet is repeatable. Names are exact and case-sensitive. Output follows caller order while keeping each sheet's full-workbook ordinal. --artifact is also repeatable. Use keys listed in the Markdown artifact table. Both options are workbook-only. brewdoc validates every request and output collision before writing.

URL mode accepts one public HTTP(S) HTML page with strict UTF-8 decoding. It uses GET, verified TLS and default ports only. Private destinations, URL credentials and HTTPS downgrades are refused; proxy environment settings are ignored. Each redirect and resource is validated. Acquisition allows at most five redirects and an 8 MiB body limit. --timeout SECONDS sets one maximum waiting budget for loading and optional JS, default 30 seconds. It accepts positive finite seconds within the supported clock range and returns earlier when ready. Increase it for pages that need longer loading or JS waiting. Bounded cleanup and local conversion can add time after waiting ends. No login, cookie profile or bot bypass is offered. Local files and default static URL capture do not execute JavaScript.

--render-js explicitly executes the selected page's scripts locally in a separate process with bounded APIs and resources. No filesystem or shell APIs are exposed, and captures do not reuse browser state. This is not a general operating-system security sandbox. It supports DOM mutations, bounded jobs/timers and guarded GET requests. Async XHR also supports the public Algolia search endpoint used by hn.algolia.com, with its original string body and no cookies or user credentials. Other POST requests remain unsupported; search redirects are refused without replay. Supported module loading uses the native engine. Temporary localStorage starts empty for each capture; accessing it marks the result partial. Each capture starts with a writable innerWidth of 1024 CSS pixels. Width changes stay within that capture. It does not provide browser layout, iframe execution, workers, WebSockets, media or browser-wide compatibility. Native JS requires a verified QuickJS public ABI on macOS ARM64 or 64-bit glibc Linux (x86_64/aarch64). Windows, macOS Intel, musl Linux and 32-bit systems refuse JS. Missing components or unsupported runtimes refuse before fetching. See URL acquisition limits.

URL receipts use brewdoc.receipt/2 with redacted acquisition provenance and the frozen snapshot identity. Timing-only expiry preserves available valid content with acquisition.capture_status=partial and a diagnostic. The receipt distinguishes the current DOM from an explicitly marked fallback to the already acquired original HTML, where JS completion is unknown. No HTML is returned from an incomplete response. Hard network, security, heap, native fault, tamper or resource failures exit 1 and preserve an existing output file. Exit 0 confirms conversion, not script completion or article fidelity. Live acquisition can change; conversion of the same frozen snapshot remains deterministic. No 90% coverage or global JavaScript compatibility is claimed.

Combining --url with a local path or --self-check, or using --render-js without --url, exits 2 before acquisition. Invalid timeout values and an explicit --timeout without --url also exit 2. URL requests with --sheet or --artifact exit 1 with an HTML refusal receipt before HTTP.

Every successful document uses brewdoc.markdown/2. A quoted source title is followed by Metadata, Artifacts, Known omissions, Contents, then anchored content units. PDF keys are page/000001, workbook keys are sheet/000001, DOCX keys are chapter/000001, PPTX keys are slide/000001, EPUB keys follow declared spine ordinals as chapter/000001, and HTML has one chapter/000001. Metadata contains deterministic source identity, selection, content hashes, conversion tallies, and artifact capabilities. It excludes paths, timestamps, permissions, host data, and unsupported author metadata. EPUB adds source-declared publication titles, creators, languages and identifiers.

For a conversion request, one JSON receipt line goes to stdout first (wrapped here, unit_keys omitted):

{"artifacts": [], "broken_ligature_words": 0, "columns_split": 1,
 "dropped": {"cid_survivors": 0, "control_chars": 0, "ligatures": 0, "nbsp": 0,
             "non_ascii_replaced": 0, "page_numbers": 3, "pua_glyphs": 0,
             "running_heads": 0, "soft_hyphens": 0},
 "file_ok": true, "markdown_schema": "brewdoc.markdown/2",
 "not_carried": ["images, figures and the text drawn inside them",
                 "a table that spans a page break",
                 "text rotated out of the horizontal reading order"],
 "out": "file.md",
 "reason": "pdf rendered: 7 pages, 1 tables, 26 text regions, 1 column splits",
 "receipt_schema": "brewdoc.receipt/1",
 "route": "pdf", "source": "nist-sp800-145.pdf", "tables": 1,
 "text_regions": 26, "unit_kind": "page", "units": 7}

Paths resolve against the current working directory; --out creates parent directories. Exit code is non-zero on refusal; the receipt line is still printed.

Formats

.xlsx and .xlsm expose original OOXML formulas as one deterministic JSON artifact per formula-bearing selected sheet. Markdown still contains cached values. .xlsm and .xlsb can expose one complete related vbaProject.bin as opaque bytes. brewdoc does not execute macros, decompile VBA, report source-module counts, evaluate formulas, or infer dependency graphs.

HTML reads local strict UTF-8, with an optional BOM. It preserves meaningful Unicode, navigation, main content, sidebars and footers in recovered DOM order. It never fetches links or resources, executes scripts, submits forms or infers interactive values. Links use a restricted scheme policy; form actions remain inert metadata. Inline code outside tables becomes a literal fenced block; surrounding emphasis and links remain balanced. Safe links in literal code and around code/images retain destination records without changing the source literals. Table-cell code uses labelled escaped text. Quotes and list items containing standalone child blocks use explicit nesting boundaries in place of visual Markdown containers. Plain list paragraphs retain their item marker and continuation indentation. See HTML scope and limits.

EPUB reads one package with static XHTML in EPUB 2 or 3. Each declared spine item becomes a chapter, including empty and non-linear items, front matter and back matter. Publication navigation retains source hierarchy and links to generated chapter and fragment anchors. Declared metadata and meaningful Unicode are preserved. Images remain inert alt/title/target records; no pixels or external resources are read. EPUB adds no parser dependency. Malformed required content, unsafe archives, unavailable navigation targets, encrypted reading text, declared fixed layout and unsupported required media without an XHTML fallback refuse the whole book before output writes. Multiple rootfiles are outside this supported profile. See EPUB scope and limits.

The public Python API exposes EPUB, HTML and presentation rendering and generic workbook artifact access:

import brewdoc

markdown, tally = brewdoc.render_epub("book.epub")
code, receipt, markdown = brewdoc.run("book.epub", out="book.md")
markdown, tally = brewdoc.render_html("page.html")
code, receipt, markdown = brewdoc.run("page.html", out="page.md")
markdown, tally = brewdoc.render_presentation("slides.pptx")
refs = brewdoc.list_book_artifacts("book.xlsm", sheets=("Calc",))
payload = brewdoc.read_book_artifact("book.xlsm", refs[0].key, sheets=("Calc",))
code, receipt, markdown = brewdoc.run(
    "book.xlsm", out="book.md", sheets=("Calc",),
    artifact_outputs={refs[0].key: "formulas.json"},
)

artifact_outputs maps artifact keys to paths. The CLI builds it from repeated --artifact KEY=PATH.

CSV and TSV are intentionally unsupported and are not planned. LLMs can read their text tables directly, so brewdoc does not convert them to Markdown.

Planned formats, excluded ones, deferred ones and the never-list: FORMATS.md. A planned format is refused by name today. The adapter contract and the ordered checklist for adding one are in docs/architecture.md.

PDF regions, in the order tried:

What it does not do

Determinism

No timestamps or unordered iteration. Two renders of one file have the same sha256; --self-check pins it. PDF, DOCX, PPTX and workbook output remains ASCII. HTML and EPUB Markdown use UTF-8 without BOM and LF newlines, with the same bytes in the API, file output and CLI stdout. Receipt JSON keeps ASCII escaping.

Size and speed

These measurements cover the existing PDF/workbook baseline before the HTML parser addition. They do not measure HTML or EPUB conversion or the new installed footprint.

Benchmarks

Browse the corpus catalog, follow the measurement method, and inspect the full-run results. Completed runs include actual outputs and known failures; capture completion does not mean content quality passed.

Releasing

Tag-triggered, see RELEASING.md.

License

MIT, see LICENSE.