TrueParse

High-performance, local-first PDF parsing and document intelligence engine that converts arbitrary PDFs into canonical, structured JSON and clean Markdown while extracting true visual assets and reconstructed table grids.

Overview

TrueParse is built for engineering teams who need accurate, sub-pixel document parsing without sending sensitive data to external cloud APIs. It parses complex multi-column layouts, extracts embedded raster images directly from PDF bytecode, reconstructs table cell grids with row/column spans, builds relational section hierarchies, and tracks exact spatial coordinates for every element.

Key Differentiators

• True Visual Asset Extraction: Extracts actual raster image streams directly without lossy re-compression or replacing images with whole-page screenshots.
• SHA-256 Deduplication: Hashes every visual asset — raster and vector crop — across pages, so a logo repeated on 40 pages is stored once with 40 recorded occurrences.
• Structural Table Reconstruction: Parses ruled and unruled tables into 2D cell grids with real row/column spans measured from the grid geometry, bounding boxes, and pre-rendered Markdown/HTML.
• N-Column Reading Order: Discovers column boundaries from whitespace gutters rather than assuming a page midline, so three-column journals and off-centre sidebars read in the right order.
• Document Graph & Heading Hierarchy: Reconstructs heading trees from a document-wide font ladder and numbering patterns (3.1.2 Results), section boundaries, parent-child containment, and caption-to-asset bindings.
• Retrieval-Ready Chunking: Emits chunks.jsonl where every chunk carries its heading breadcrumb, page range, and bounding boxes — enough provenance to cite a RAG answer back to a rectangle on a page.
• Optional Local OCR: Scanned pages are recognised via RapidOCR/ONNX — a pure wheel, no system packages — keeping the zero-cloud guarantee intact.
• Multi-Worker Parallel Processing: SQLite-backed background task manager with process-based workers, real-time page-by-page progress, and multi-file batch execution.
• Zero Cloud Dependencies: 100% local, privacy-preserving, and free of recurring per-page API costs.

4 Deployment & Usage Modes

TrueParse is engineered to support four distinct integration patterns:

1. Pip Package (CLI) — Install directly from PyPI into your environment.
2. Direct Python SDK — Embed directly into your application codebase.
3. FastAPI REST Service — Self-host as a standalone microservice with Swagger documentation.
4. Docker Container — One-command local container hosting with zero Python setup.

1. Pip Package & CLI

Install TrueParse directly via pip:

# Install from PyPI
pip install trueparse

# With local OCR support for scanned documents
pip install "trueparse[ocr]"
# On a minimal Linux image, OCR also needs OpenCV's system libraries:
#   sudo apt-get install -y libgl1 libglib2.0-0

# Or install latest directly from GitHub
pip install git+https://github.com/RITURAJRAMAN/TrueParse.git

This installs the core library and registers the trueparse command-line utility:

CLI Commands

# 1. Parse a PDF document (outputs to data/output/{document_id}/)
trueparse parse "path/to/document.pdf"

# 2. Parse with custom output directory and page limits
trueparse parse "path/to/document.pdf" -o "./results" --max-pages 5

# 3. Choose a parsing profile (fast | balanced | accurate | maximum_accuracy)
trueparse parse "path/to/document.pdf" --profile accurate

# 4. Emit RAG-ready chunks alongside the JSON
trueparse parse "path/to/document.pdf" --chunks --chunk-size 512 --overlap 64

# 5. Emit additional export formats
trueparse parse "path/to/document.pdf" --html --text

# 6. Force OCR on a scanned document (needs: pip install "trueparse[ocr]")
trueparse parse "path/to/scan.pdf" --ocr always

# 7. Unlock an encrypted PDF
trueparse parse "path/to/protected.pdf" --password "s3cret"

# 8. Re-chunk an already-parsed document without re-parsing the PDF
trueparse chunk "data/output/doc_abc123/output/document.json" --chunk-size 256

# 9. Asynchronous parsing with live progress spinner
trueparse parse-async "path/to/document.pdf"

# 10. Parallel batch parsing across all PDFs in a directory
trueparse batch "path/to/pdf_folder" --recursive

# 11. Instant PDF forensics and structure inspection (no full parse)
trueparse inspect "path/to/document.pdf"

# 12. Show the installed version
trueparse --version

Run trueparse <command> --help for the full flag list. Chunking flags (--chunks, --chunk-strategy, --chunk-size, --overlap) accept section, token, or hybrid for the strategy.

2. Python SDK

Import trueparse directly in your Python applications with top-level exports:

from trueparse import PDFParser, ParseOptions, ParsingProfile, OCRMode

# 1. Configure parser options
options = ParseOptions(
    profile=ParsingProfile.BALANCED,
    extract_images=True,
    extract_tables=True,
    extract_charts=True,
    extract_formulas=True,
    ocr=OCRMode.AUTO,        # auto | always | never
    password=None,           # for encrypted PDFs
    emit_chunks=True,        # also write chunks.jsonl
    debug=False,
)

# 2. Initialize parser instance
parser = PDFParser(options=options)

# 3. Parse PDF document
doc = parser.parse("path/to/document.pdf")

# 4. Access document metadata and quality metrics
print(f"Document ID: {doc.id}")
print(f"Page Count: {len(doc.pages)}")
print(f"Unique Extracted Assets: {len(doc.assets)}")
print(f"Quality Score: {doc.quality.overall_score:.2f}")

# 5. Traverse structured sections and heading hierarchy
for section in doc.sections:
    print(f"[{section.level}] {section.title} (Pages: {section.page_range})")

# 6. Access page elements with exact coordinates
for page in doc.pages:
    print(f"\n--- Page {page.page_number} ({page.width}x{page.height} pt) ---")
    for elem in page.elements:
        bbox = f"({elem.bbox.x0:.1f}, {elem.bbox.y0:.1f}, {elem.bbox.x1:.1f}, {elem.bbox.y1:.1f})"
        print(f"[{elem.type}] {bbox}: {elem.content[:60]}")

Retrieval Chunking

Chunks carry the provenance needed to cite a retrieved answer back to its exact location in the source PDF:

from trueparse import DocumentChunker, ChunkStrategy

chunks = DocumentChunker.chunk(
    doc,
    strategy=ChunkStrategy.HYBRID,   # section | token | hybrid
    max_tokens=512,
    overlap_tokens=64,
)

for chunk in chunks:
    print(f"[{chunk.id}] {' > '.join(chunk.section_path)}")
    print(f"  pages {chunk.page_start}-{chunk.page_end} | ~{chunk.token_estimate} tokens")
    print(f"  elements: {chunk.element_ids}")
    print(f"  text: {chunk.text[:80]}...")

# Serialize to newline-delimited JSON for your vector store
jsonl = DocumentChunker.to_jsonl(chunks)

Tables are never split across chunks, and paragraphs larger than the budget are split on sentence boundaries.

3. FastAPI REST Service

Clone and host the TrueParse service as a high-throughput REST API with live background worker queues:

# 1. Clone repository and install dependencies
git clone https://github.com/RITURAJRAMAN/TrueParse.git
cd TrueParse

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\Activate.ps1
pip install -e ".[dev]"

# 2. Launch FastAPI service
uvicorn trueparse.api.routes:app --host 0.0.0.0 --port 8000 --reload

• Interactive Swagger UI: http://localhost:8000/docs
• ReDoc Documentation: http://localhost:8000/redoc

REST API Endpoints

Server Configuration

The service is configured by environment variables, not by request parameters. Output location in particular is server-controlled: a client cannot choose where files are written.

Exposing the service beyond localhost? Set TRUEPARSE_API_KEY. The API is unauthenticated by default so that local use needs no configuration.

export TRUEPARSE_API_KEY="your-secret-key"
export TRUEPARSE_OUTPUT_ROOT="/var/lib/trueparse/output"
trueparse serve --host 0.0.0.0 --port 8000

curl -X POST "http://localhost:8000/v1/documents/parse" \
  -H "X-API-Key: your-secret-key" \
  -F "file=@document.pdf"

Example cURL Requests

Synchronous Parse:

curl -X POST "http://localhost:8000/v1/documents/parse" \
  -F "file=@document.pdf" \
  -F "extract_images=true" \
  -F "extract_tables=true" \
  -F "emit_chunks=true" \
  -F "profile=accurate"

Forensic Inspection (upload, no full parse):

curl -X POST "http://localhost:8000/v1/documents/inspect" \
  -F "file=@document.pdf"

Asynchronous Background Parse & Progress Polling:

# Submit document (returns job_id immediately)
curl -X POST "http://localhost:8000/v1/documents/parse-async" \
  -F "file=@large_document.pdf"

# Poll live progress
curl "http://localhost:8000/v1/documents/jobs/{job_id}"

Parallel Multi-File Batch Parse:

curl -X POST "http://localhost:8000/v1/batches/parse-async" \
  -F "files=@document_1.pdf" \
  -F "files=@document_2.pdf" \
  -F "files=@document_3.pdf"

# Check batch aggregation
curl "http://localhost:8000/v1/batches/{batch_id}"

4. Docker Container Deployment

Host TrueParse locally with Docker without needing Python or local environment setup:

Pull Pre-Built Image (GitHub Container Registry)

# 1. Pull the official pre-built multi-platform container image
docker pull ghcr.io/riturajraman/trueparse:latest

# 2. Run with mounted local output volume
docker run -d \
  -p 8000:8000 \
  -v $(pwd)/data/output:/app/data/output \
  --name trueparse-api \
  ghcr.io/riturajraman/trueparse:latest

The published image includes local OCR support (~1.35 GB). To build a slimmer image without it (~263 MB), at the cost of scanned-document support:

docker build --build-arg INSTALL_OCR=false -t trueparse-slim .

Check which variant you are running with GET /health — it reports "ocr_available": true|false.

Set server configuration with -e, for example:

docker run -d -p 8000:8000 \
  -e TRUEPARSE_API_KEY="your-secret-key" \
  -v $(pwd)/data/output:/app/data/output \
  ghcr.io/riturajraman/trueparse:latest

Using Docker Compose (Build Locally)

# Start container with persistent data volume mounts
docker compose up -d --build

The service is available immediately at http://localhost:8000/docs. Parsed outputs will automatically be saved to ./data/output/ on your host machine.

Using Docker CLI (Build Locally)

# 1. Build Docker image
docker build -t trueparse .

# 2. Run container with mounted output volume
docker run -d \
  -p 8000:8000 \
  -v $(pwd)/data/output:/app/data/output \
  --name trueparse-service \
  trueparse

Standardized Workspace Output

Every parsed document generates a structured, self-contained workspace under data/output/{document_id}/:

data/output/{document_id}/
├── source/
│   └── {filename}.pdf        # Archived copy of the source PDF
├── assets/
│   ├── images/               # Deduplicated raster images (asset_img_0001.jpeg)
│   ├── figures/              # Isolated vector drawings & figures (asset_fig_0001.png)
│   ├── charts/               # Detected chart assets (asset_chart_0001.png)
│   ├── diagrams/             # Diagram assets (asset_diagram_0001.png)
│   └── formulas/             # Formula assets (asset_formula_0001.png)
├── output/
│   ├── document.json         # Canonical structured document graph
│   ├── document.md           # Derived clean Markdown export
│   ├── document.html         # Self-contained HTML export (--html)
│   ├── document.txt          # Plain reading-order text (--text)
│   └── chunks.jsonl          # Retrieval-ready chunks (--chunks)
├── intermediate/             # Intermediate processing artifacts
└── debug/
    └── pages/                # Page render debug artifacts (optional)

Canonical JSON Schema Structure

The output document.json conforms to a strict, typed schema:

{
  "id": "doc_f76e021893a5",
  "schema_version": "1.1",
  "engine_version": "0.1.2",
  "source_file": "annual_report.pdf",
  "metadata": {
    "title": "Annual Financial Report",
    "page_count": 106,
    "sha256": "f76e021893a5..."
  },
  "pages": [
    {
      "page_number": 1,
      "width": 612.0,
      "height": 792.0,
      "elements": [
        {
          "id": "elem_p0001_0001",
          "type": "heading_1",
          "page": 1,
          "bbox": { "x0": 54.0, "y0": 72.0, "x1": 450.0, "y1": 96.0 },
          "reading_order": 1,
          "content": "Executive Summary",
          "confidence": 0.99,
          "provenance": {
            "method": "native_pdf",
            "engine": "pymupdf",
            "confidence": 0.99
          }
        }
      ],
      "asset_ids": ["asset_img_0001"]
    }
  ],
  "sections": [
    {
      "id": "sec_0001",
      "title": "Executive Summary",
      "level": 1,
      "page_range": [1, 3],
      "element_ids": ["elem_p0001_0001", "elem_p0001_0002"],
      "subsections": []
    }
  ],
  "assets": {
    "asset_img_0001": {
      "id": "asset_img_0001",
      "type": "image",
      "relative_path": "assets/images/asset_img_0001.jpeg",
      "format": "jpeg",
      "width": 800,
      "height": 600,
      "sha256": "3a7b1c...",
      "occurrences": [
        { "page": 1, "bbox": { "x0": 54.0, "y0": 120.0, "x1": 558.0, "y1": 340.0 } }
      ]
    }
  },
  "relationships": [
    {
      "source_id": "elem_p0001_0003",
      "target_id": "asset_img_0001",
      "type": "caption_for"
    }
  ],
  "quality": {
    "overall_score": 0.97,
    "text_score": 0.99,
    "layout_score": 0.96,
    "table_score": 0.94,
    "coverage_score": 0.41,
    "ocr_pages": 0,
    "warnings": []
  }
}

Chunk Records (chunks.jsonl)

{
  "id": "chunk_00007",
  "document_id": "doc_f76e021893a5",
  "chunk_index": 7,
  "text": "Revenue for the period grew 12% year over year...",
  "token_estimate": 384,
  "section_id": "sec_0003",
  "section_path": ["Financial Review", "Revenue"],
  "page_start": 12,
  "page_end": 13,
  "bboxes": [{ "page": 12, "bbox": [54.0, 220.0, 558.0, 410.0] }],
  "element_ids": ["elem_p0012_0004", "elem_p0013_0001"],
  "element_types": ["paragraph"],
  "asset_ids": []
}

Parsing Profiles Matrix

TrueParse includes 4 configurable parsing profiles:

Profiles are not labels — each resolves to concrete engine tuning applied on every page.

OCR: Auto means only pages detected as scanned are recognised, and only when the ocr extra is installed. Set --ocr always to force it, or --ocr never to disable. render_dpi overrides the profile's DPI when set explicitly.

Testing

Run the automated test suite with pytest:

# Run all tests (149 tests, fully self-contained - fixtures are generated at runtime)
pytest

# Run a specific area
pytest tests/test_security.py     # path containment, auth, upload limits
pytest tests/test_chunking.py     # RAG chunking
pytest tests/test_layout.py       # reading order, headings, paragraph merging
pytest tests/test_tables.py       # span reconstruction, cross-page merging
pytest tests/test_queue.py        # SQLite queue and worker pool

# OCR tests skip automatically unless the extra is installed
pip install "trueparse[ocr]" && pytest tests/test_ocr.py

# Lint and type-check (both gated in CI)
ruff check src tests
mypy

Documentation

• api_documentation.md — Comprehensive API & Schema Specification
• CHANGELOG.md — Release History & Changelog

License

This project is licensed under the MIT License.