pdf-jsonl-smith

pdf-jsonl-smith converts PDF-derived text into compact BigQuery-ready newline-delimited JSON.

The local app is a thin, boring wrapper around PyMuPDF for born-digital PDFs. The docai app runs Google Document AI Form Parser, but still emits compact purpose-built records instead of full Document AI payloads.

Install

uv pip install -e ".[dev]"

For the Google Document AI app:

uv pip install -e ".[dev,gcp]"

CLI

pdf-jsonl-smith local input.pdf -o text_units.jsonl

Optional metadata:

pdf-jsonl-smith local input.pdf \
  --output text_units.jsonl \
  --source-uri gs://bucket/input.pdf \
  --schema-version 0.1

By default, the CLI prints a short completion summary to stdout:

Wrote 42 records to text_units.jsonl from input.pdf.

Use --quiet to suppress the summary.

The older pdf2jsonl command remains available as a compatibility alias for the local PyMuPDF extractor:

pdf2jsonl input.pdf -o text_units.jsonl

The docai app uploads a local PDF to Cloud Storage, runs Google Document AI, and writes compact JSONL records:

pdf-jsonl-smith docai input.pdf \
  --output compact.jsonl \
  --project my-project \
  --location us \
  --processor-id PROCESSOR_ID \
  --staging-uri gs://bucket/pdf-jsonl-smith/input/

By default, docai emits compact text, key-values, tables, and lists records. Use --include to narrow the output:

pdf-jsonl-smith docai input.pdf \
  --output compact.jsonl \
  --project my-project \
  --location us \
  --processor-id PROCESSOR_ID \
  --staging-uri gs://bucket/pdf-jsonl-smith/input/ \
  --include text,key-values

Use --jsonl-gcs-uri to upload the compact JSONL output after writing it locally.

For a beginner-friendly explanation of Document AI, buckets, and authentication, see the Document AI page in the published docs.

The bq app is reserved for the BigQuery workflow:

pdf-jsonl-smith bq

The docai app policy is to extract from rich Google Document AI payloads, but emit only compact, purpose-built records. Full Document AI payloads are not part of the final JSONL output.

Useful jq checks:

jq -r '.unit_type' compact.jsonl | sort | uniq -c

jq -r 'select(.unit_type=="text") | .text' compact.jsonl

jq -r 'select(.unit_type=="key_value") | [.key, .value, .confidence] | @tsv' \
  compact.jsonl

Because the output is one unified JSONL stream, filter by unit_type when you want only main text, form key-value pairs, table cells, or list items.

Python API

from pdf_jsonl_smith import extract_pdf_to_jsonl

count = extract_pdf_to_jsonl(
    input_path="input.pdf",
    output_path="text_units.jsonl",
    source_uri="gs://bucket/input.pdf",
)

The function returns the number of JSONL records written.

Output

The output is UTF-8 JSONL: one standalone JSON object per text unit. Records are flat and BigQuery-friendly.

For the local extractor, extraction starts from PyMuPDF text blocks. Adjacent blocks on the same page may be merged when their geometry looks like wrapped lines from the same paragraph. This is intentionally conservative and does not try to infer document semantics.

Fields:

schema_version STRING
doc_id STRING
source_uri STRING
filename STRING
page_number INT64
unit_type STRING
unit_index INT64
text STRING
bbox_x0 FLOAT64
bbox_y0 FLOAT64
bbox_x1 FLOAT64
bbox_y1 FLOAT64
char_count INT64
word_count INT64
extractor STRING
extracted_at TIMESTAMP
warning STRING

Supported

• Born-digital PDFs.
• PyMuPDF block-based text extraction.
• Conservative same-page wrapped-line merging.
• Google Document AI Form Parser compact output.
• UTF-8 JSONL output.
• BigQuery-friendly flat records.
• One unified JSONL stream with unit_type filters.

Not Supported

For the local extractor:

• Scanned PDFs / OCR.
• Forms.
• Image text.

For all extractors:

• Full Document AI payload mirroring.
• Perfect reading order.
• Cross-page paragraph reconstruction.
• Semantic heading detection.
• Lossless table reconstruction.

Development

uv sync --dev
uv run pytest
uv run ruff check
uv run sphinx-build -b html docs/source docs/build/html
uv build

Project Infrastructure

This project is maintained as a uv-based Python package.

Releases use Python Semantic Release with zero-version releases enabled for the initial 0.x series. The GitHub Actions release workflow runs on main, creates releases from conventional commits, builds distributions with uv, and publishes to PyPI using trusted publishing.

Documentation sources live in docs/source, build output goes under docs/build, and the site uses Sphinx with the Furo theme. It is intended to be built by Read the Docs through the RTD GitHub App integration.

License

MIT