pdf-jsonl-smith 0.1.0

Convert born-digital PDF text blocks to BigQuery-ready JSONL.

pdf-jsonl-smith

pdf-jsonl-smith is a thin, boring wrapper around PyMuPDF for converting text-based PDF files into BigQuery-ready newline-delimited JSON.

It is intended for upstream ELT pipelines that need flat text-unit records, not document understanding.

Install

uv pip install -e ".[dev]"

CLI

pdf2jsonl input.pdf -o text_units.jsonl

Optional metadata:

pdf2jsonl input.pdf \
  --output text_units.jsonl \
  --source-uri gs://bucket/input.pdf \
  --schema-version 0.1

By default, the CLI prints a short completion summary to stdout:

Wrote 42 records to text_units.jsonl from input.pdf.

Use --quiet to suppress the summary.

Python API

from pdf_jsonl_smith import extract_pdf_to_jsonl

count = extract_pdf_to_jsonl(
    input_path="input.pdf",
    output_path="text_units.jsonl",
    source_uri="gs://bucket/input.pdf",
)

The function returns the number of JSONL records written.

Output

The output is UTF-8 JSONL: one standalone JSON object per text unit. Records are flat and BigQuery-friendly.

For v0.1, extraction starts from PyMuPDF text blocks. Adjacent blocks on the same page may be merged when their geometry looks like wrapped lines from the same paragraph. This is intentionally conservative and does not try to infer document semantics.

Fields:

schema_version STRING
doc_id STRING
source_uri STRING
filename STRING
page_number INT64
unit_type STRING
unit_index INT64
text STRING
bbox_x0 FLOAT64
bbox_y0 FLOAT64
bbox_x1 FLOAT64
bbox_y1 FLOAT64
char_count INT64
word_count INT64
extractor STRING
extracted_at TIMESTAMP
warning STRING

Supported

• Born-digital PDFs.
• PyMuPDF block-based text extraction.
• Conservative same-page wrapped-line merging.
• UTF-8 JSONL output.
• BigQuery-friendly flat records.
• Images ignored.

Not Supported

• Scanned PDFs / OCR.
• Table reconstruction.
• Perfect reading order.
• Cross-page paragraph reconstruction.
• Semantic heading detection.
• Forms.
• Image text.

Development

uv sync --dev
uv run pytest
uv run ruff check
uv run sphinx-build -b html docs/source docs/build/html
uv build

Project Infrastructure

This project is maintained as a uv-based Python package.

Releases use Python Semantic Release with zero-version releases enabled for the initial 0.x series. The GitHub Actions release workflow runs on main, creates releases from conventional commits, builds distributions with uv, and publishes to PyPI using trusted publishing.

Documentation sources live in docs/source, build output goes under docs/build, and the site uses Sphinx with the Furo theme. It is intended to be built by Read the Docs through the RTD GitHub App integration.

License

MIT