Enterprise-grade SDK for document ingestion, OCR, semantic chunking, and RAG-ready processing

These details have not been verified by PyPI

Project description

OmniDoc Python SDK

Enterprise-grade document intelligence for Agentic AI, RAG pipelines, and automation systems

Python 3.9+ v0.4.0 Apache 2.0 40+ formats OCR

Overview
Installation
Supported Document Types
Architecture
Quick Start
Core API — extract_pdf()
Universal Loader — load_document()
Output Formats
OCR Engines
RAG Pipeline
PII Masking
Enterprise Configuration
Structured Logging
Retry & Resilience
CLI
Optional Extras
Contributing & Development
Changelog

Overview

OmniDoc converts raw documents — PDFs, spreadsheets, slides, emails, scanned images, notebooks, source code, archives, audio/video, and more — into clean, structured, agent-ready outputs.

Layer	What it does
Detection	40+ file types detected automatically from extension
Extraction	Text, tables, layout blocks, metadata per document
Normalization	Headings, bullets, metrics, slide-to-report restructuring
Serialization	Python `Document`, JSON, or agent-native TOON

RAG pipeline? Install omnidoc-rag for intent-aware chunking, confidence scoring, streaming, graph linking, evaluation, and vector DB adapters.

Installation

Minimal (PDF + local OCR)

pip install omnidoc-sdk

With Office documents (Word, Excel, PowerPoint)

pip install "omnidoc-sdk[office,data]"

With web, email, and structured formats

pip install "omnidoc-sdk[docs,email,yaml,ebook,openoffice,legacy]"

With code & notebooks

pip install "omnidoc-sdk[code]"

With cloud OCR (AWS Textract / Azure Form Recognizer)

pip install "omnidoc-sdk[cloud-ocr]"

With audio/video transcription (Whisper)

pip install "omnidoc-sdk[media]"

Full enterprise install (everything)

pip install "omnidoc-sdk[all]"

System dependencies

OS	Command
macOS	`brew install poppler tesseract`
Ubuntu / Debian	`apt-get install poppler-utils tesseract-ocr libgl1`
RHEL / CentOS	`yum install poppler-utils tesseract`

Supported Document Types

Every type below is fully supported by load_document(). Optional extras are loaded lazily — only installed when the specific file type is used.

Category	Extensions	Extractor	Install Extra	Capabilities
PDF	`.pdf`	Full pipeline	core	Digital, scanned, slides; OCR, layout ML, tables, VLM
Plain Text	`.txt`	`TextExtractor`	core	Auto-detects encoding (UTF-8 → latin-1 → chardet fallback)
Markdown / RST	`.md` `.rst`	`MarkdownExtractor`	core	Headings, code blocks, lists preserved as sections
Word	`.docx`	`DocxExtractor`	`office`	Paragraphs, headings, inline tables
Excel / Spreadsheet	`.xlsx` `.xls` `.csv` `.tsv`	`SpreadsheetExtractor`	`office, data`	Multi-sheet; multi-encoding CSV/TSV; headers + typed rows
PowerPoint	`.pptx`	`PPTXExtractor`	`office`	Slide title + body text per slide
OpenDocument	`.odt` `.ods` `.odp`	`OpenDocumentExtractor`	`openoffice`	LibreOffice Writer, Calc, Impress — text, tables, slides
RTF	`.rtf`	`RTFExtractor`	`legacy`	Legacy Word / WordPad format, full control-word stripping
Images (OCR)	`.png` `.jpg` `.jpeg` `.tiff` `.tif` `.webp` `.heic` `.heif` `.bmp` `.gif`	`ImageExtractor`	core (`image` for HEIC)	Tesseract OCR, per-image timeout, HEIC/HEIF via pillow-heif
SVG	`.svg`	`HTMLExtractor`	`docs`	Text nodes extracted, markup stripped
HTML / Web	`.html` `.htm`	`HTMLExtractor`	`docs`	Script/style removed; per-block sections (p, h1–h6, li, td)
JSON	`.json`	`JSONExtractor`	core	Flattened key-value pairs as document sections
XML	`.xml`	`XMLExtractor`	core	Element text + tail content extracted as sections
YAML	`.yaml` `.yml`	`YAMLExtractor`	`yaml`	Parsed with PyYAML, serialised to pretty JSON for chunking
E-Book	`.epub`	`EPUBExtractor`	`ebook`	Per-chapter sections, author + title metadata
Email — EML	`.eml`	`EMLExtractor`	core (stdlib)	Headers (From/To/Subject/Date), plain-text + HTML parts, attachment list
Email — Outlook	`.msg`	`MSGExtractor`	`email`	Sender, recipients, subject, body, attachment list via extract-msg
Jupyter Notebook	`.ipynb`	`NotebookExtractor`	`code`	Markdown + code + output cells as sections; kernel/language metadata
Source Code	`.py` `.java` `.js` `.ts` `.go` `.cpp` `.c` `.rs`	`CodeExtractor`	core (`code` for token count)	Block-level splitting (function/class boundaries), language metadata, optional Pygments token count
Archives	`.zip` `.tar` `.tar.gz` `.tar.bz2` `.tar.xz` `.7z`	`ArchiveExtractor`	core (`archive` for .7z)	Path-traversal guard (zip-slip), 512 MB per-member bomb limit; returns extracted file paths
Audio	`.mp3` `.wav` `.m4a`	`MediaExtractor`	`media`	Whisper transcription, segment-level sections, auto language detection
Video	`.mp4` `.mov` `.avi` `.mkv`	`MediaExtractor`	`media`	Audio extracted via ffmpeg, then Whisper transcription

Architecture

Input File
    │
    ├─ detect_document_type()          ← 40+ extensions, compound (.tar.gz)
    │
    ├─ PDF Pipeline ────────────────────────────────────────────────────
    │   ├─ is_scanned_pdf()            ← digital vs. scanned detection
    │   ├─ extract_text()              ← pdfplumber / pypdf
    │   ├─ convert_from_path()         ← pdf2image rasterisation
    │   ├─ detect_layout()             ← Detectron2 layout ML (optional)
    │   ├─ order_blocks()              ← column-aware reading order
    │   ├─ OCR Engine                  ← Tesseract / Textract / Azure
    │   ├─ extract_tables()            ← camelot lattice extraction
    │   ├─ extract_ocr_tables()        ← table detection from images
    │   └─ normalize_slides()          ← slide → report restructuring
    │
    ├─ Non-PDF Extractors ──────────────────────────────────────────────
    │   ├─ TextExtractor               ← .txt
    │   ├─ MarkdownExtractor           ← .md / .rst
    │   ├─ DocxExtractor               ← .docx
    │   ├─ SpreadsheetExtractor        ← .xlsx / .xls / .csv / .tsv
    │   ├─ PPTXExtractor               ← .pptx
    │   ├─ OpenDocumentExtractor       ← .odt / .ods / .odp
    │   ├─ RTFExtractor                ← .rtf
    │   ├─ HTMLExtractor               ← .html / .htm / .svg
    │   ├─ JSONExtractor               ← .json
    │   ├─ XMLExtractor                ← .xml
    │   ├─ YAMLExtractor               ← .yaml / .yml
    │   ├─ EPUBExtractor               ← .epub
    │   ├─ EMLExtractor                ← .eml (stdlib, zero deps)
    │   ├─ MSGExtractor                ← .msg (Outlook)
    │   ├─ ImageExtractor              ← .png / .jpg / .tiff / .webp / .heic ...
    │   ├─ NotebookExtractor           ← .ipynb
    │   ├─ CodeExtractor               ← .py / .java / .js / .ts / .go / ...
    │   ├─ ArchiveExtractor            ← .zip / .tar / .tar.gz / .7z
    │   └─ MediaExtractor              ← .mp3 / .mp4 / .wav / .mov ...
    │
    ├─ Post-processing ─────────────────────────────────────────────────
    │   ├─ clean_text()
    │   ├─ normalize_slide_sections()
    │   ├─ build_document()            ← assembles Document dataclass
    │   └─ mask_pii()                  ← Presidio (optional)
    │
    └─ Output ──────────────────────────────────────────────────────────
        ├─ Document                    ← Python dataclass
        ├─ JSON                        ← RAG-ready dict
        └─ TOON                        ← agent-native format

Quick Start

from omnidoc.loader.load import load_document

# Works for every supported file type
doc = load_document("report.pdf")

print(doc.raw_text[:500])         # cleaned full text
print(doc.metadata)               # file info, page count, pipeline

# Pass to omnidoc-rag for chunking, evaluation, and vector DB ingestion

Core API — `extract_pdf()`

The PDF-specific pipeline with full control over every stage.

from omnidoc.pdf.pipeline import extract_pdf

doc = extract_pdf(
    path="document.pdf",          # required: path to PDF
    enable_layout=True,           # layout ML for reading-order (default: True)
    enable_cloud_ocr=False,       # use AWS Textract for scanned PDFs
    enable_vlm=False,             # use Donut VLM for complex layouts
    enable_pii_masking=False,     # redact PII via Presidio
    output_format="document",     # "document" | "json" | "toon"
    config=None,                  # OmnidocConfig (uses default when None)
)

Parameter reference

Parameter	Type	Default	Description
`path`	str	—	Absolute or relative path to a PDF file. Raises `FileNotFoundError` if missing.
`enable_layout`	bool	`True`	Use layout ML for correct column / multi-column reading order.
`enable_cloud_ocr`	bool	`False`	Route scanned PDFs through AWS Textract. Requires `[cloud-ocr]`.
`enable_vlm`	bool	`False`	Use Donut VLM for complex scanned layouts. Requires `[vlm]`.
`enable_pii_masking`	bool	`False`	Redact PII (names, emails, SSNs). Requires `[privacy]`.
`output_format`	str	`"document"`	`"document"` → `Document`; `"json"` → dict; `"toon"` → agent format.
`config`	OmnidocConfig	`None`	Override DPI, file size limit, OCR timeout, retries, and more.

Example — digital PDF

doc = extract_pdf("annual_report.pdf", enable_layout=True)

# Per-page sections
for section in doc.sections:
    print(f"Page {section.page}: {section.text[:200]}")

# Extracted tables
for table in doc.tables:
    print(f"Page {table.page}: headers={table.headers}")
    for row in table.rows[:3]:
        print(" ", row)

Example — scanned PDF with cloud OCR

import os
os.environ["AWS_DEFAULT_REGION"] = "us-east-1"

doc = extract_pdf("scanned_invoice.pdf", enable_cloud_ocr=True)
print(doc.raw_text)

Universal Loader — `load_document()`

Accepts any supported file type. Extension is detected automatically.

from omnidoc.loader.load import load_document

doc = load_document(
    path="file.ext",
    config=None,   # optional OmnidocConfig
)

Text, Markdown & Code

# Plain text — multi-encoding fallback (UTF-8 → latin-1 → chardet)
doc = load_document("notes.txt")
print(doc.raw_text)

# Markdown
doc = load_document("spec.md")
for sec in doc.sections:
    print(sec.text[:100])

# Python source — splits at function/class boundaries
doc = load_document("main.py")
for sec in doc.sections:
    print(f"Block: {sec.text[:80]}")

# Other code files — .java, .js, .ts, .go, .cpp, .c, .rs
doc = load_document("Main.java")
print(doc.metadata["language"])   # "java"
print(doc.metadata["lines"])      # line count

Office Documents

# Word
doc = load_document("contract.docx")
print(doc.sections[0].text)

# PowerPoint
doc = load_document("deck.pptx")
for i, sec in enumerate(doc.sections, 1):
    print(f"Slide {i}: {sec.text[:80]}")

Spreadsheets

# Excel
doc = load_document("financials.xlsx")
for table in doc.tables:
    print("Headers:", table.headers)
    print("Row 1:", table.rows[0])

# CSV (multi-encoding detection)
doc = load_document("users.csv")

# TSV
doc = load_document("export.tsv")

Presentations

doc = load_document("strategy.pptx")
for i, sec in enumerate(doc.sections, 1):
    print(f"Slide {i}: {sec.text[:100]}")

OpenDocument (LibreOffice)

# Writer document
doc = load_document("report.odt")
print(doc.raw_text)

# Calc spreadsheet — returns tables
doc = load_document("data.ods")
for table in doc.tables:
    print("Sheet:", table.headers)

# Impress presentation
doc = load_document("slides.odp")
for sec in doc.sections:
    print(sec.text[:100])

Web & Markup

# HTML — script/style stripped, per-block sections
doc = load_document("page.html")
print(doc.metadata.get("title"))   # <title> tag value
print(doc.raw_text[:500])

# SVG — text nodes extracted
doc = load_document("diagram.svg")

Structured Data — JSON, XML, YAML

# JSON
doc = load_document("config.json")
print(doc.raw_text)          # flattened key-value representation

# XML
doc = load_document("data.xml")

# YAML
doc = load_document("settings.yaml")
print(doc.raw_text)          # pretty-printed JSON representation of the YAML

# .yml alias also works
doc = load_document(".github/workflows/ci.yml")

Images (OCR)

# OCR via Tesseract
doc = load_document("screenshot.png")
print(doc.raw_text)

# TIFF, WebP, BMP, GIF — all handled
doc = load_document("scan.tiff")

# HEIC / HEIF (iPhone photos) — requires [image] extra
doc = load_document("photo.heic")

# Language hint for Tesseract
from omnidoc.extractors.image.extractor import ImageExtractor
doc = ImageExtractor(lang="eng+fra", timeout=120).extract("french_doc.jpg")

E-Books (EPUB)

# Each chapter becomes a Section
doc = load_document("book.epub")
print(doc.metadata["title"])
print(doc.metadata["authors"])

for i, sec in enumerate(doc.sections, 1):
    print(f"Chapter {i}: {sec.text[:100]}")

Email (.eml / .msg)

# RFC-822 email — zero extra dependencies
doc = load_document("message.eml")
print(doc.metadata["subject"])
print(doc.metadata["from"])
print(doc.metadata["to"])
print(doc.metadata.get("attachments", []))
print(doc.raw_text)

# Outlook .msg — requires [email] extra
doc = load_document("outlook_message.msg")
print(doc.metadata["subject"])
print(doc.raw_text)

Legacy — RTF

# Rich Text Format — requires [legacy] extra
doc = load_document("document.rtf")
print(doc.raw_text)

Jupyter Notebooks

# Markdown + code + outputs all extracted
doc = load_document("analysis.ipynb")
print(doc.metadata["kernel"])     # e.g. "Python 3 (ipykernel)"
print(doc.metadata["language"])   # "python"

for sec in doc.sections:
    print(sec.text[:150])          # one section per cell

Audio & Video (Transcription)

# Audio — requires [media] extra (openai-whisper)
doc = load_document("podcast.mp3")
print(doc.metadata["language"])        # auto-detected language
for sec in doc.sections:
    print(sec.text)                    # one section per Whisper segment

# Video — audio extracted via ffmpeg, then transcribed
doc = load_document("meeting.mp4")
print(doc.raw_text)

# Custom model size and language
from omnidoc.extractors.media.extractor import MediaExtractor
doc = MediaExtractor(model="medium", language="en").extract("lecture.wav")

Output Formats

Document Object

The default Python dataclass output.

from omnidoc.core.document import Document, Section, Table

doc: Document = load_document("report.pdf")

# Metadata
print(doc.metadata)
# {
#   "file": "report.pdf",
#   "pages": 12,
#   "source_type": "pdf",
#   "pipeline": "pdf",
#   "scanned": False,
# }

# Full text
print(doc.raw_text)

# Per-page sections
for section in doc.sections:
    print(f"  Page {section.page}: {section.text[:80]}")

# Tables
for table in doc.tables:
    print(f"  Page {table.page}: {table.headers}")
    for row in table.rows:
        print("    ", row)

JSON Output

from omnidoc.pdf.pipeline import extract_pdf

result = extract_pdf("report.pdf", output_format="json")

# result is a dict
print(result["metadata"])
print(result["sections"][0])
print(result["tables"])
print(result["raw_text"][:200])

TOON Output

result = extract_pdf("report.pdf", output_format="toon")

# TOON is Anthropic's agent-native structured format
print(result["type"])
print(result["content"])

OCR Engines

Tesseract (local)

from omnidoc.ocr.tesseract import TesseractOCR

ocr = TesseractOCR(
    dpi=300,       # rendering DPI (higher = better accuracy, slower)
    timeout=60,    # per-page timeout in seconds (SIGALRM on Unix)
    lang="eng",    # Tesseract language code(s), e.g. "eng+fra"
)

AWS Textract

from omnidoc.ocr.aws_textract import TextractOCR

ocr = TextractOCR(
    region="us-east-1",
    timeout=120,
    max_retries=3,
    retry_backoff=2.0,
)

Azure Form Recognizer

from omnidoc.ocr.azure_form_recognizer import AzureFormRecognizer

ocr = AzureFormRecognizer(
    endpoint="https://my-resource.cognitiveservices.azure.com/",
    api_key="your-key",
    timeout=120,
    max_retries=3,
    model_id="prebuilt-read",   # or "prebuilt-layout" for tables
)

RAG Pipeline

Semantic chunking, streaming, evaluation, graph linking, and vector DB adapters live in the companion package omnidoc-rag:

pip install omnidoc-rag

from omnidoc.loader.load import load_document
from omnidoc_rag.chunker import chunk_document
from omnidoc_rag.evaluation import evaluate_rag_result
from omnidoc_rag.vectordb.chroma import ChromaAdapter

doc = load_document("investor_deck.pdf")

# Intent-aware semantic chunks
chunks = chunk_document(doc)
for c in chunks:
    print(f"[{c.intent}] {c.confidence:.2f} — {c.text[:100]}")

# Evaluate a retrieval result
score = evaluate_rag_result(
    query="What was the revenue growth?",
    answer="Revenue grew 24% YoY.",
    chunks=chunks,
)
print(score["overall"], score["verdict"])

# Upsert to ChromaDB
adapter = ChromaAdapter(collection_name="reports")
adapter.upsert(chunks)

See omnidoc-rag documentation for the full API including streaming, graph linking, cross-document stitching, Pinecone/Weaviate/pgvector adapters, and LangChain/LlamaIndex integration.

PII Masking

from omnidoc.pdf.pipeline import extract_pdf

doc = extract_pdf("contract.pdf", enable_pii_masking=True)

# Names, emails, phone numbers, SSNs, credit cards replaced with <ENTITY_TYPE>
print(doc.raw_text)
# "The contract was signed by <PERSON> on behalf of <ORGANIZATION>."

Requires pip install "omnidoc-sdk[privacy]".

Enterprise Configuration

Override defaults via OmnidocConfig or environment variables.

from omnidoc.config import OmnidocConfig

cfg = OmnidocConfig(
    max_file_mb=200,        # OMNIDOC_MAX_FILE_MB    (default 100)
    pdf_dpi=300,            # OMNIDOC_PDF_DPI        (default 250)
    ocr_timeout=120,        # OMNIDOC_OCR_TIMEOUT    (default 60)
    max_retries=5,          # OMNIDOC_MAX_RETRIES    (default 3)
    retry_backoff=2.0,      # OMNIDOC_RETRY_BACKOFF  (default 2.0)
    aws_region="us-east-1", # OMNIDOC_AWS_REGION
    enable_pii_masking=True,
    log_level="INFO",       # OMNIDOC_LOG_LEVEL
)

doc = load_document("large_report.pdf", config=cfg)

Or set environment variables before importing:

export OMNIDOC_MAX_FILE_MB=500
export OMNIDOC_OCR_TIMEOUT=180
export OMNIDOC_LOG_LEVEL=DEBUG

Structured Logging

from omnidoc.utils.logging_config import configure_logging

# Human-readable (default)
configure_logging(level="INFO")

# JSON logging for log aggregators (Datadog, CloudWatch, Splunk)
configure_logging(level="INFO", json=True)

JSON output example:

{"ts": "2024-01-15T10:23:41.123Z", "level": "INFO", "logger": "omnidoc.loader.load", "msg": "load_document: path='report.pdf' size=1240.5KB"}
{"ts": "2024-01-15T10:23:42.456Z", "level": "DEBUG", "logger": "omnidoc.pdf.pipeline", "msg": "extract_pdf: 12 pages, scanned=False"}

Retry & Resilience

Apply automatic retry to any function that calls a flaky external service.

from omnidoc.utils.retry import with_retry

@with_retry(
    max_attempts=3,
    base_delay=1.0,
    backoff_factor=2.0,
    jitter=True,
    exceptions=(TimeoutError, ConnectionError),
)
def call_my_api(doc_bytes: bytes):
    ...

CLI

# Extract any supported file — print JSON to stdout
omnidoc extract report.pdf

# With format control
omnidoc extract report.pdf --format json
omnidoc extract report.pdf --format toon

# With cloud OCR
omnidoc extract scanned.pdf --cloud-ocr aws

# Mask PII in output
omnidoc extract contract.pdf --pii

# Debug logging
omnidoc extract report.pdf --log-level DEBUG

# Extract non-PDF files
omnidoc extract data.xlsx
omnidoc extract notebook.ipynb
omnidoc extract message.eml
omnidoc extract podcast.mp3

Optional Extras

Install only what you need. All heavy dependencies are loaded lazily at runtime.

Extra	Install	Unlocks
`office`	`pip install "omnidoc-sdk[office]"`	`.docx`, `.xlsx`, `.xls`, `.pptx`
`data`	`pip install "omnidoc-sdk[data]"`	pandas-backed CSV/TSV/XLSX with full type inference
`docs`	`pip install "omnidoc-sdk[docs]"`	`.html`, `.htm`, `.svg`, `.xml`
`yaml`	`pip install "omnidoc-sdk[yaml]"`	`.yaml`, `.yml`
`ebook`	`pip install "omnidoc-sdk[ebook]"`	`.epub`
`openoffice`	`pip install "omnidoc-sdk[openoffice]"`	`.odt`, `.ods`, `.odp`
`legacy`	`pip install "omnidoc-sdk[legacy]"`	`.rtf`
`email`	`pip install "omnidoc-sdk[email]"`	`.msg` (Outlook) — `.eml` needs no extra
`image`	`pip install "omnidoc-sdk[image]"`	HEIC / HEIF image formats (iPhone photos)
`code`	`pip install "omnidoc-sdk[code]"`	Jupyter notebooks (`.ipynb`) + Pygments token count
`archive`	`pip install "omnidoc-sdk[archive]"`	`.7z` archives via py7zr
`media`	`pip install "omnidoc-sdk[media]"`	`.mp3`, `.mp4`, `.wav`, `.m4a`, `.mov`, `.avi`, `.mkv`
`cloud-ocr`	`pip install "omnidoc-sdk[cloud-ocr]"`	AWS Textract + Azure Form Recognizer
`layout`	`pip install "omnidoc-sdk[layout]"`	Detectron2 layout ML for complex PDFs
`vlm`	`pip install "omnidoc-sdk[vlm]"`	Donut vision-language model for scanned PDFs
`privacy`	`pip install "omnidoc-sdk[privacy]"`	PII masking via Microsoft Presidio
`all`	`pip install "omnidoc-sdk[all]"`	Everything above

Contributing & Development

Setup

git clone https://github.com/your-org/omnidoc-python-sdk.git
cd omnidoc-python-sdk
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

# Editable install with all extras
pip install -e ".[all]"

# Dev tools
pip install build twine pytest pytest-cov ruff black mypy

Verify:

python -c "import omnidoc; print('OK')"
omnidoc extract --help

System dependencies:

OS	Command
macOS	`brew install poppler tesseract`
Ubuntu/Debian	`apt-get install poppler-utils tesseract-ocr libgl1`
RHEL/CentOS	`yum install poppler-utils tesseract`

Project Structure

omnidoc-python-sdk/
├── omnidoc/
│   ├── __init__.py              # Public API
│   ├── config.py                # OmnidocConfig (env-var driven)
│   ├── version.py
│   ├── adapters/                # LangChain, LlamaIndex thin wrappers
│   ├── cli/                     # omnidoc CLI entry point
│   ├── core/                    # Document, Section, Table dataclasses
│   ├── extractors/
│   │   ├── archive/             # .zip / .tar / .7z
│   │   ├── code/                # .py / .java / .js / .ts / .go / .cpp / .rs
│   │   ├── ebook/               # .epub (ebooklib)
│   │   ├── email/               # .eml (stdlib) + .msg (extract-msg)
│   │   ├── image/               # .png / .jpg / .tiff / .webp / .heic …
│   │   ├── legacy/              # .rtf (striprtf)
│   │   ├── markdown/            # .md / .rst
│   │   ├── media/               # .mp3 / .mp4 / .wav (Whisper)
│   │   ├── notebook/            # .ipynb (nbformat)
│   │   ├── office/              # .docx / .xlsx / .pptx
│   │   ├── openoffice/          # .odt / .ods / .odp (odfpy)
│   │   ├── structured/          # .json / .xml / .yaml
│   │   ├── text/                # .txt
│   │   └── web/                 # .html / .htm / .svg (beautifulsoup4)
│   ├── layout/                  # Layout ML, block ordering
│   ├── loader/
│   │   ├── load.py              # Universal loader (40+ types)
│   │   ├── router.py            # detect_document_type()
│   │   └── types.py             # DocumentType enum
│   ├── ocr/                     # Tesseract, Textract, Azure engines
│   ├── pdf/                     # PDF pipeline, detector, normaliser
│   ├── postprocess/             # Slide normaliser, heading detection
│   ├── privacy/                 # PII masking
│   ├── tests/                   # Unit + smoke tests
│   └── utils/                   # text, retry, logging, serialize
├── testing/                     # Manual integration scripts
├── pyproject.toml
└── README.md

Running Tests

# All tests
pytest omnidoc/tests/ -v

# Single file
pytest omnidoc/tests/test_text.py -v
pytest omnidoc/tests/test_new_extractors.py -v

# With coverage
pytest omnidoc/tests/ --cov=omnidoc --cov-report=term-missing

# Skip slow/cloud tests
pytest omnidoc/tests/ -v -m "not slow and not cloud"

Which tests need which extras:

Test file	Required extra
`test_text.py`	none
`test_spreadsheet.py`	`data`
`test_pdf.py`	none (pdfplumber)
`test_archive.py`	none / `archive` for .7z
`test_image.py`	none (Pillow)
`test_docx.py`	`office`
`test_config.py`	none
`test_new_extractors.py`	`docs`, `code`, `ebook`, `openoffice` (skipped if absent)

Lint and type-check:

ruff check omnidoc/
black --check omnidoc/
mypy omnidoc/ --ignore-missing-imports

# Auto-fix
black omnidoc/
ruff check omnidoc/ --fix

Building & Publishing

Build:

rm -rf dist/ build/ omnidoc_sdk.egg-info/
python -m build
twine check dist/*

Test on TestPyPI first:

twine upload --repository testpypi dist/*

# Verify install
pip install \
  --index-url https://test.pypi.org/simple/ \
  --extra-index-url https://pypi.org/simple/ \
  omnidoc-sdk

Publish to PyPI:

twine upload dist/*
pip install omnidoc-sdk==0.4.0

Credentials — ~/.pypirc:

[distutils]
index-servers = pypi testpypi

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-YOUR_TEST_TOKEN

[pypi]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-YOUR_PROD_TOKEN

chmod 600 ~/.pypirc

GitHub Actions CI/CD — store PYPI_API_TOKEN and TEST_PYPI_API_TOKEN as repository secrets. The .github/workflows/ci.yml publishes to TestPyPI on merge to main and to PyPI on a version tag (git tag v0.4.0 && git push origin v0.4.0).

Versioning

Version is defined once in pyproject.toml. Follow Semantic Versioning:

Change	Example	Bump
Bug fix	Fix path traversal guard	`0.4.0 → 0.4.1`
New feature	Add new extractor	`0.4.0 → 0.5.0`
Breaking change	Rename public API	`0.4.0 → 1.0.0`

PyPI does not allow re-uploading the same version. Always bump before rebuilding.

Release Checklist

[ ] ruff check omnidoc/           — zero errors
[ ] black --check omnidoc/        — no formatting changes
[ ] pytest omnidoc/tests/ -v      — all pass
[ ] Version bumped in pyproject.toml
[ ] Changelog updated below
[ ] rm -rf dist/ && python -m build
[ ] twine check dist/*            — both artifacts PASSED
[ ] TestPyPI round-trip verified
[ ] twine upload dist/*           — production upload
[ ] git tag vX.Y.Z && git push origin vX.Y.Z

Troubleshooting

HTTPError: 400 Bad Request — File already exists — PyPI does not allow overwriting. Bump version, rebuild, re-upload.

twine check fails with "description failed to render" — README contains unsupported HTML. Test with:

pip install readme-renderer[md]
python -m readme_renderer README.md -o /tmp/preview.html

403 Forbidden on upload — Token scope wrong (use Entire account for first upload), or 2FA not enabled on your PyPI account.

ModuleNotFoundError for an optional dependency — Install the relevant extra:

pip install "omnidoc-sdk[ebook,openoffice,email,legacy,yaml,media]"

Changelog

[0.4.0] — 2026-04-10

Added

omnidoc/config.py — OmnidocConfig dataclass; all settings driven by OMNIDOC_* env vars
omnidoc/utils/retry.py — @with_retry decorator with exponential backoff and jitter
omnidoc/utils/logging_config.py — structured JSON or plain-text logging under omnidoc.*
omnidoc/ocr/azure_form_recognizer.py — AzureFormRecognizer OCR engine (was missing)
omnidoc/extractors/web/html_extractor.py — HTMLExtractor (bs4, per-block sections)
omnidoc/extractors/structured/yaml_extractor.py — YAMLExtractor
omnidoc/extractors/email/eml_extractor.py — EMLExtractor (stdlib, zero deps)
omnidoc/extractors/email/msg_extractor.py — MSGExtractor (Outlook via extract-msg)
omnidoc/extractors/ebook/epub_extractor.py — EPUBExtractor (ebooklib)
omnidoc/extractors/openoffice/extractor.py — OpenDocumentExtractor (.odt/.ods/.odp)
omnidoc/extractors/legacy/rtf_extractor.py — RTFExtractor (striprtf)
omnidoc/extractors/notebook/extractor.py — NotebookExtractor (.ipynb)
omnidoc/extractors/code/extractor.py — CodeExtractor (15 languages, block splitting)
omnidoc/extractors/media/extractor.py — MediaExtractor (Whisper, ffmpeg)
Full CI pipeline in .github/workflows/ci.yml

Changed

omnidoc/pdf/pipeline.py — input validation, per-step error isolation, structured logging
omnidoc/loader/load.py — 40+ types wired; validation, file-size check, config passthrough
omnidoc/ocr/aws_textract.py — @with_retry, 10 MB limit enforced, correct confidence averaging
omnidoc/ocr/tesseract.py — SIGALRM-based per-page timeout, dpi/lang constructor args
omnidoc/extractors/image/extractor.py — corrupt-image detection, colour-mode normalisation
omnidoc/extractors/office/docx.py — lazy import, corrupt-file guard, safe style access
omnidoc/extractors/office/pptx.py — is identity check for title shape (was ==, crashed)
omnidoc/extractors/office/spreadsheet.py — multi-encoding CSV, fillna(""), sections= fix
omnidoc/extractors/archive/extractor.py — zip-slip guard, 512 MB bomb limit, temp-dir cleanup
omnidoc/extractors/structured/json_xml.py — XML text/tail content now extracted correctly
omnidoc/utils/serialize.py — reuses doc.chunks; handles both dataclass and dict chunks
RAG/streaming pipeline extracted to standalone omnidoc-rag package

Fixed

SpreadsheetExtractor crashed with TypeError (missing sections= argument)
ArchiveExtractor leaked temp directories on every call
ArchiveExtractor vulnerable to zip-slip path traversal
TextExtractor hard-coded UTF-8 causing UnicodeDecodeError on latin-1 files
PPTXExtractor AttributeError when title placeholder absent
XMLExtractor discarded all text node content
.gitignore contained wrong AL/Dynamics-365 template

[0.3.9] — 2025-12-01

Initial release: PDF extraction, Tesseract/Textract OCR, layout ML, semantic chunking, TOON/JSON output, LangChain/LlamaIndex adapters, PII masking, streaming, CLI. Supported: PDF, TXT, MD, DOCX, XLSX, CSV, PPTX, PNG, JPG, JSON, XML, ZIP.

OmniDoc Python SDK · v0.4.0 · Apache 2.0 · 40+ document formats · RAG pipeline → omnidoc-rag

Project details

These details have not been verified by PyPI

Release history Release notifications | RSS feed

0.4.2

May 15, 2026

0.4.1

May 14, 2026

This version

0.4.0

Apr 10, 2026

0.3.9

Dec 29, 2025

0.2.6

Dec 29, 2025

0.2.5

Dec 29, 2025

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

omnidoc_sdk-0.4.0.tar.gz (125.1 kB view details)

Uploaded Apr 10, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

omnidoc_sdk-0.4.0-py3-none-any.whl (130.5 kB view details)

Uploaded Apr 10, 2026 Python 3

File details

Details for the file omnidoc_sdk-0.4.0.tar.gz.

File metadata

Download URL: omnidoc_sdk-0.4.0.tar.gz
Upload date: Apr 10, 2026
Size: 125.1 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/6.2.0 CPython/3.12.8

File hashes

Hashes for omnidoc_sdk-0.4.0.tar.gz
Algorithm	Hash digest
SHA256	`bd4adc6a809c9465e1ec1a4b164ed6beedcabadbd12eed38de8ef612bf270686`
MD5	`d6a3aff8d266765dc0e76addafea1a53`
BLAKE2b-256	`071a46d6ce6c6f7c7bad738624fbc778d19dde82f8b5b6aa11ae272ff76d2eac`

See more details on using hashes here.

File details

Details for the file omnidoc_sdk-0.4.0-py3-none-any.whl.

File metadata

Download URL: omnidoc_sdk-0.4.0-py3-none-any.whl
Upload date: Apr 10, 2026
Size: 130.5 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/6.2.0 CPython/3.12.8

File hashes

Hashes for omnidoc_sdk-0.4.0-py3-none-any.whl
Algorithm	Hash digest
SHA256	`7409aca3ddc499d4deb87dbaf1aaf4bb10911ccb144bab9f48457f58a07326ec`
MD5	`072d9c2e16f87dec7a9924407af773d5`
BLAKE2b-256	`31aab5392a9d3c50d4249edfb5214fda31cd16aede902e9cedd84c8648adafde`

See more details on using hashes here.

omnidoc-sdk 0.4.0

Navigation

Verified details

Maintainers

Unverified details

Meta

Classifiers

Project description

OmniDoc Python SDK

Table of Contents

Overview

Installation

Minimal (PDF + local OCR)

With Office documents (Word, Excel, PowerPoint)

With web, email, and structured formats

With code & notebooks

With cloud OCR (AWS Textract / Azure Form Recognizer)

With audio/video transcription (Whisper)

Full enterprise install (everything)

System dependencies

Supported Document Types

Architecture

Quick Start

Core API — extract_pdf()

Parameter reference

Example — digital PDF

Example — scanned PDF with cloud OCR

Universal Loader — load_document()

Text, Markdown & Code

Office Documents

Spreadsheets

Presentations

OpenDocument (LibreOffice)

Web & Markup

Structured Data — JSON, XML, YAML

Images (OCR)

E-Books (EPUB)

Email (.eml / .msg)

Legacy — RTF

Jupyter Notebooks

Archives

Audio & Video (Transcription)

Output Formats

Document Object

JSON Output

TOON Output

OCR Engines

Tesseract (local)

AWS Textract

Azure Form Recognizer

RAG Pipeline

PII Masking

Enterprise Configuration

Structured Logging

Retry & Resilience

CLI

Optional Extras

Contributing & Development

Setup

Project Structure

Running Tests

Building & Publishing

Versioning

Release Checklist

Troubleshooting

Changelog

[0.4.0] — 2026-04-10

Added

Changed

Fixed

[0.3.9] — 2025-12-01

Project details

Verified details

Maintainers

Unverified details

Meta

Classifiers

Release history Release notifications | RSS feed

Download files

Source Distribution

Core API — `extract_pdf()`

Universal Loader — `load_document()`