PageLedger: auditable OCR and document extraction runs

PageLedger records document extraction one page at a time. It calls your chosen engine, tracks source files and adapter settings, checks budgets, and writes the text with its provenance and review queue. Runs are directories of JSON, YAML, Markdown, and raw output that you can inspect without a service or database.

It grew out of a Soviet census digitization project. The same records help archives, historians, and research teams trace an extracted page back to its source and reconstruct how it was produced.

Install

PageLedger requires Python 3.10 or later:

pip install pageledger
pageledger --version

For PDFs, install "pageledger[pdf]". Scanned PDFs also need Poppler and Tesseract installed separately. pageledger doctor checks the available tools.

First run

The built-in text adapter needs no OCR engine or provider. This example creates two source pages; the replacement character on page 2 deliberately triggers a review warning.

printf 'first page\fsecond page with a replacement character �\n' > sample.txt
pageledger init-config --out pageledger.yml
pageledger run sample.txt --config pageledger.yml --out runs/first
pageledger inspect-run runs/first
sed -n '1,20p' runs/first/raw/doc_0001_page_0002.txt
sed -n '1,80p' runs/first/audit.md
pageledger verify-run runs/first

runs/first/ contains the extracted text, per-page provenance, quality signals, cost evidence, and an executable rerun plan. Unknown cost stays unknown. Verification checks the ledger's integrity; review the output against the source to judge transcription accuracy.

Rerun the flagged pages into a new directory and compare the results:

pageledger rerun runs/first --config pageledger.yml --out runs/second
pageledger compare-runs runs/first runs/second

This example uses the same adapter, so its output stays the same. Supply a stronger adapter config when needed. A rerun retains the parent linkage and source page numbers; you decide which outputs to use.

The text tutorial continues through CSV export, review notes and bundle replay. For scans, follow the PDF/OCR tutorial.

Choose an engine

Built-in adapters read plain text files (text), PDF text layers (pdf_text), scans with Tesseract (pdf_ocr) or RapidOCR (rapidocr, after pip install 'pageledger[rapidocr]'), and page images with a vision model behind an OpenAI-compatible endpoint, on your machine or hosted (vision). The engine recipes show how to set each up, with Tesseract's model for pre-1918 Russian print and Apple Vision, and what each did well and badly on 24 transcribed historical pages. OCR options compares the approaches, and the adapter protocol connects any other engine.

Process a document

Use process when one job should manage a whole document: it reads the text layer, sends pages with defects to OCR and, if you allow it, to a vision model, and keeps every attempt. Create processing.yml:

schema_version: "0.1"
processing:
  local_text:
    adapter: pdf_text
  local_ocr:
    adapter: pdf_ocr
    adapter_options:
      lang: eng
      dpi: 300
  limits:
    max_attempt_pages: 100
    max_image_pages: 0

With the PDF extra, Poppler and Tesseract installed, sample a document:

pageledger process book.pdf --config processing.yml --pages "1-10" --out jobs/book
pageledger inspect-job jobs/book
pageledger verify-job jobs/book

jobs/book/report.md lists first the pages that need a person, with the text the job selected, whether engines agree and any review so far. Review those pages in a spreadsheet with pageledger review-sheet, record the decisions with review-job, and write the text out as plain text, Markdown, JSONL or TEI with pageledger export. See document processing, warnings and holds and export. The document-job tutorial walks through a job from a source checkout, and process a collection through a folder of real documents.

Document jobs and resume need POSIX file locks: use macOS, Linux or, on Windows, WSL. Plain run uses no such locks but is not tested on Windows. The tutorials use bash.

Recover interrupted work

A document job keeps recovery records automatically. For an individual run, opt in when starting it:

pageledger run book.pdf --adapter pdf_ocr --resumable --out runs/book
# After an interruption:
pageledger resume runs/book

Use pageledger resume jobs/book for a document job. Resume reuses saved work and never repeats a request that may already have been processed. See the checkpoint specification for run recovery rules and the processing specification for document-job recovery.

What the ledger records

Core depends only on PyYAML; the PDF extra adds pypdf with AES support. The capabilities and limits say what is and is not supported, and the glossary defines the terms.

Documentation

Contributing

Testing a collection we haven't seen? Open a corpus report with the script, adapter, page count and a redacted sample. New collections are how the quality signals improve. Development setup and guidelines are in CONTRIBUTING.md.

Citing

Cite the software with CITATION.cff. PageLedger keeps software and source-data citations separate: dataset_citation in the config records the latter in every manifest.

MIT license.