Blackletter

A reference to blackletter law, this tool removes potentially copyrighted material from legal case law PDFs. This fulfills our goal of respecting any intellectual property rights others may have while making it possible to digitize and publish case law. This is essential to our mission of making case law accessible to all — a prerequisite for meaningful participation in our democracy.

Proprietary annotations removed from judicial opinions include headnotes, captions, and key cites.

Installation

pip install blackletter

The base install covers opinion pairing, redaction, margin and page-number helpers, and OCR — everything that runs without a detection/OCR model. The heavy inference frameworks live in optional extras, so a consumer that offloads detection (e.g. to a remote GPU worker) can keep a lean install:

Notes:

• [analyze] includes [detect] — the analyze pipeline runs YOLO in addition to PaddleOCR.
• Without [refine], docTR headnote refinement is unavailable, so redaction helpers must be called with skip_doctr=True. [refine] is kept separate from [detect]/[analyze] so consumers that skip refinement (e.g. a web/daemon image) save ~200 MB of dependencies.
• The process and draw CLI commands run YOLO and require [detect]; process also runs the refinement pass and requires [refine].

Or install from source:

git clone https://github.com/freelawproject/blackletter
cd blackletter
pip install -e '.[analyze,refine]'

Quick Start

Command line:

blackletter process path/to/volume.pdf --reporter f3d --volume 952 --first-page 1 --output output/

Python:

from blackletter import process

process(
    "path/to/volume.pdf",
    "output/",
    reporter="f3d",
    volume="952",
    first_page=1,
)

This runs the full pipeline: YOLO detection, page number extraction, opinion splitting, and redaction, all in one pass. No text layer is added: the geometry measures the page's ink directly. Pass --text-layer if you want the delivered PDFs to be searchable.

How It Works

The process command runs a single-pass pipeline:

1. Detection: Runs a YOLO model to identify proprietary elements (headnotes, captions, key cites, brackets, etc.) and structural elements (page numbers, dividers, footnotes)
2. Page Numbers: Extracts and validates page numbers using OCR on detected regions
3. Opinion Pairing: Matches case captions to key icons to identify opinion boundaries
4. Splitting & Redaction: Produces per-opinion variants and an optional per-page LLM split:
   • Unredacted (opt-in, via --unredacted): Raw opinion pages extracted from the source
   • Redacted: Per-opinion PDFs with potentially copyrighted content (headnotes, brackets, key icons) blacked out
   • LLM (opt-in, via --llm): One PDF per source page, sliced from the fully redacted document, with an invisible <--CASEEND--> marker stamped on every redacted Key-icon location so downstream LLM passes can detect opinion boundaries

A searchable text layer is optional and comes last, over the files that have already been redacted and masked (--text-layer on the CLI, or api.add_text_layer from Python). Doing it this way avoids spending OCR time on content that is about to be blacked out. To OCR the source before detection instead, pass --ocr.

Additionally produces:

• A full redacted copy of the entire document
• Extracted case law images (charts, photos, etc.) as PNGs
• A detections.json export of all YOLO detections for review tooling

Models

Blackletter uses three YOLO models, selected via CLI flags:

The models are hosted at freelawproject/blackletter-weights and are downloaded automatically to blackletter/weights/ on first use, keeping the package itself small.

Command Line Options

Process Command

blackletter process PDF [OPTIONS]

Positional Arguments:
  pdf                       Path to the source PDF

Options:
  --reporter STR            Reporter abbreviation (e.g. f3d, a3d)
  --volume STR              Volume number
  --first-page INT          Page number of the first page in the PDF (default: 1)
  -o, --output PATH         Base output directory (required)
  --model PATH              Path to custom YOLO model weights
  --medium                  Use the medium model (17 classes)
  --large                   Use the large model (21 classes)
  --footnotes               Extract footnotes into separate PDFs
  --unredacted              Also generate unredacted opinion PDFs
  --llm                     Also generate per-page LLM PDFs with <--CASEEND--> stamps
  --no-shrink               Skip downsampling (default: shrink to ~148 KB/page)
  --optimize {0,1,2,3}      ocrmypdf optimization level (default: 1)
  --bitonal                 Convert to 1-bit B&W before processing (for already-bitonal scans)
  --detect-only             Stop after detection and pairing — no PDFs written (Phase 1 only)
  --text-layer              Add a searchable text layer to the generated PDFs, after redaction
  --ocr                     OCR the source before detection (rarely needed; prefer --text-layer)

Validate Command

QA tool that checks a PDF's page number sequence for missing, duplicate, or misnumbered pages. Uses YOLO to locate page number regions, then PaddleOCR to read them, with Tesseract and GLM-OCR as fallbacks.

blackletter validate path/to/volume.pdf
blackletter validate path/to/volume.pdf --first-page 100 --last-page 500
blackletter validate path/to/volume.pdf --json

If the filename follows the convention reporter.volume.first.last.pdf (e.g. sct.143.1.888.pdf), the expected page range is inferred automatically.

Features:

• Parallel OCR across multiple workers
• Auto-correction of consistent OCR misreadings (e.g. systematic off-by-800 errors)
• Detection of gaps, duplicates, backwards jumps, and page ranges (e.g. "31-32")
• Structural checks for blank pages and orientation changes

Requires optional dependencies: pip install blackletter[analyze]

Draw Command

Visualize YOLO detections on a PDF — useful for debugging model output:

blackletter draw path/to/volume.pdf --output annotated.pdf
blackletter draw path/to/volume.pdf --output annotated.pdf --labels CASE_CAPTION KEY_ICON HEADNOTE
blackletter draw path/to/volume.pdf --output annotated.pdf --large

Output Structure

output/<reporter>/<volume>/<first-page>/
    <reporter>.<volume>.<first>.<last>.pdf   # OCR'd/processed source PDF
    <reporter>.<volume>.redacted.pdf         # Full redacted document

    detections.json       # All YOLO detections (label, bbox, confidence per page)
    pages_meta.json       # Column bounds and midpoints per page
    opinions.json         # Opinion pairs with outside-opinion rects
    redaction_rects.json  # Precomputed redaction rectangles (used by review UI)
    margin_rects.json     # Margin cleanup rectangles

    images/               # Extracted case law images (PNGs)
    unredacted/           # Individual opinion PDFs (raw, no redaction) — only if --unredacted
    redacted/             # Individual opinion PDFs (copyrighted content redacted)
    llm/                  # Per-page fully-redacted PDFs with invisible <--CASEEND-->
                          # stamps on each Key-icon location — only if --llm

The JSON files are designed for use with a review UI — they allow manual inspection and adjustment of detections and redaction boundaries before final output is committed.

Detection Labels

Labels detected across all models (availability depends on model size):

Margin Cleanup

After redaction, Blackletter automatically white-outs scan artifacts in page margins using the PDF text layer to find content boundaries. Pages with narrow text spans (appendices, image pages) are skipped automatically.

Requirements

• Python 3.12+
• Tesseract OCR (for image-only PDFs)
• libgl1 (Linux only): the base install uses non-headless opencv-python, which loads libGL.so.1 when cv2 is imported. Minimal images (e.g. python:3.12-slim, distroless) don't ship it by default.

Install tesseract:

# macOS
brew install tesseract

# Ubuntu/Debian
sudo apt install tesseract-ocr

Install libgl1 (Linux):

# Ubuntu/Debian
sudo apt install libgl1

License

GNU Affero General Public License v3

Contributing

Contributions welcome!