PageLedger: auditable OCR and document extraction runs

PageLedger records document extraction one page at a time. It calls your chosen engine, tracks source files and adapter settings, checks budgets, and writes the text with its provenance and review queue. Runs are directories of JSON, YAML, Markdown, and raw output that you can inspect without a service or database.

It grew out of a Soviet census digitization project. The same records help archives, historians, and research teams trace an extracted page back to its source and reconstruct how it was produced.

Install

PageLedger 0.5.0 requires Python 3.10 or later:

pip install pageledger==0.5.0
pageledger --version

For PDFs, install "pageledger[pdf]==0.5.0". Scanned PDFs also need Poppler and Tesseract installed separately. pageledger doctor checks the available tools.

First run

The built-in text adapter needs no OCR engine or provider. This example creates two source pages; the replacement character on page 2 deliberately triggers a review warning.

printf 'first page\fsecond page with a replacement character �\n' > sample.txt
pageledger init-config --out pageledger.yml
pageledger run sample.txt --config pageledger.yml --out runs/first
pageledger inspect-run runs/first
sed -n '1,20p' runs/first/raw/doc_0001_page_0002.txt
sed -n '1,80p' runs/first/audit.md
pageledger verify-run runs/first

runs/first/ contains the extracted text, per-page provenance, quality signals, cost evidence, and an executable rerun plan. Unknown cost stays unknown. Verification checks the ledger's integrity; review the output against the source to judge transcription accuracy.

Rerun the flagged pages into a new directory and compare the results:

pageledger rerun runs/first --config pageledger.yml --out runs/second
pageledger compare-runs runs/first runs/second

This example uses the same adapter, so its output stays the same. Supply a stronger adapter config when needed. A rerun retains the parent linkage and source page numbers; you decide which outputs to use.

The text tutorial continues through CSV export, review notes, and bundle replay. For scans, follow the PDF/OCR tutorial.

Process a document

Document processing and resumable runs require a POSIX system such as macOS or Linux. Use process when you want one job to manage local text extraction, OCR for pages with defect evidence, and optional image-model attempts. The job retains every attempt and publishes a transcript with links to source pages and a report of unresolved work.

Create processing.yml:

schema_version: "0.1"
run:
  adapter: pdf_text
processing:
  local_ocr:
    adapter: pdf_ocr
    adapter_options:
      lang: eng
      dpi: 300
  limits:
    max_attempt_pages: 100
    max_image_pages: 0

With the PDF extra, Poppler, and Tesseract installed, sample a document:

pageledger process book.pdf --config processing.yml --pages "1-10" --out jobs/book
pageledger inspect-job jobs/book
pageledger verify-job jobs/book

Open jobs/book/transcript.md for the selected text and jobs/book/report.md for the source links, attempts, and review reasons. Image stages require an explicit adapter and a positive page limit. A completed job can still have pages awaiting human review; review-job records decisions bound to the source and selected output. See the document processing guide for configuration, budgets, and review receipts.

Recover interrupted work

Document jobs retain recovery evidence automatically. For an individual run, opt in when starting it:

pageledger run book.pdf --adapter pdf_ocr --resumable --out runs/book
# After an interruption:
pageledger resume runs/book

Use pageledger resume jobs/book for a document job. Keep the source files, output directory, and adapter environment in place. Resume verifies saved responses and reuses them without another extraction call. A request started without a saved outcome stops recovery because it may already have been processed. See the checkpoint specification for supported states and recovery limits.

What the ledger records

Built-in adapters cover text files, PDF text layers, and Tesseract OCR. The adapter protocol supports other engines; the Docling example supplies local layout and VLM conversion. Core depends on PyYAML, with pypdf in the PDF extra.

classify produces a separate route map for review before run --routes. align can revise structured records without re-extracting. Bundle replay requires a locally available compatible adapter; whole document jobs and runs with image-input evidence cannot yet be bundled. The capabilities and limits describe these contracts in full.

Documentation

Contributing

Testing a collection we haven't seen? Open a corpus report with the script, adapter, page count, and a redacted sample. New collections are how the quality signals improve. Development setup and guidelines are in CONTRIBUTING.md.

Citing

Software citation lives in CITATION.cff. PageLedger keeps software and source-data citations separate; dataset_citation in the config records the latter into every manifest.

MIT license.