doc-intel-kit

Try the no-key interactive showcase

doc-intel-kit is a composable Python 3.11 toolkit for turning documents into shared, typed representations. It connects text-layer parsing, Tesseract OCR, PDF table extraction, spreadsheet ingestion, schema-driven field extraction, optional Claude tasks, and small-scale semantic search without hiding the underlying backends.

The distribution name is doc-intel-kit; the Python import remains docintel_kit.

Why use it?

Document workflows often glue together libraries that disagree about pages, coordinates, tables, and metadata. This project provides common Pydantic models such as Document, ParseResult, OcrResult, and Table, while keeping every capability independently callable and exposing registration hooks for parser, OCR, layout, extraction, LLM, and vector-store backends. That makes it useful for prototypes, evaluations, and as a typed integration layer—not as a claim of one-click, production-grade document automation.

Why this toolkit? It gives developers a composable, local-first Python integration layer with shared typed models across parsing, OCR, tables, extraction, optional LLM tasks, and small-scale RAG—without requiring a cloud document platform or hiding each backend behind one automatic pipeline.

Best for: teams combining local document libraries, projects that need inspectable intermediate results, privacy-sensitive prototypes, custom backend evaluations, and applications that want one typed contract while retaining control of routing, persistence, quality thresholds, and infrastructure.

How it compares

This is a scope comparison, not a claim that the projects are interchangeable or that doc-intel-kit is universally better.

Not a replacement for: production ingestion connectors, managed autoscaling, human-review queues, mature layout models, enterprise governance, persistent vector infrastructure, or service-level guarantees. Docling and Unstructured are broader open-source systems; LlamaParse and hyperscaler services are more complete managed offerings. doc-intel-kit is intentionally a transparent integration toolkit, and its current layout, first-page extraction, and in-memory RAG limitations are documented below.

Comparison content is paraphrased from the linked official documentation for clarity and licensing compliance. Capabilities and commercial offerings evolve; verify current vendor documentation before choosing a production platform.

Capability status

EasyOCR and PaddleOCR backends are not implemented. The registry interfaces make alternative backends possible, but installing another OCR library alone does not register it.

Architecture and real execution flow

The library exposes building blocks; your application chooses and orchestrates the branches. It does not automatically run every stage end to end.

Input path / bytes
       |
       +--> parse_document() ------> ParseResult --------+--> your application
       |                                                 |
       +--> run_ocr() -------------> OcrResult           +--> index_documents()
       |        (images/scans)                           |       |
       +--> parse_spreadsheet() ----> list[Table] --------+       v
       |                                                 |   in-memory vectors
       +--> extract_tables_from_document() -> list[Table]+       |
       |                                                         v
       +--> extract_fields() -> ExtractionResult             search_documents()

Parsed text or Tables --explicit call--> Claude task --network + API key--> result

Parsing does not automatically OCR empty pages. OCR does not automatically feed extraction. RAG indexing parses native text and does not OCR scans. Compose those steps explicitly when your workflow needs them.

Installation

Python 3.11 is required. The initial release intentionally does not claim Python 3.12+ support.

pip install doc-intel-kit

For the optional Java-backed Tabula path:

pip install "doc-intel-kit[tabula]"

For a source checkout:

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"

System dependencies

Some Python packages wrap local executables that pip cannot install:

• Tesseract OCR: required by run_ocr() and the default extraction path.
• Poppler: required to rasterize PDF pages for OCR, layout, and structured extraction.
• Ghostscript: required by Camelot's lattice table path on common installations.
• Java JRE: required only for method="tabula".
• Detectron2 plus compatible local weights: required by the default LayoutParser backend; Detectron2 is not distributed by this package.

On macOS, the common binaries are available with brew install tesseract poppler ghostscript. On Debian/Ubuntu, use sudo apt-get install tesseract-ocr poppler-utils ghostscript.

Quickstart

from docintel_kit import parse_document
from docintel_kit.spreadsheet import parse_spreadsheet

# Fast native-text path: no API key and no model download.
parsed = parse_document("report.pdf")
print(parsed.get_page_text(0))

# Spreadsheet ingestion uses the same Table model as PDF extraction.
tables = parse_spreadsheet("forecast.xlsx")
print(tables[0].headers)
print(tables[0].to_dataframe().head())

OCR and table extraction are explicit branches:

from docintel_kit import run_ocr, extract_tables_from_document

ocr = run_ocr("scanned-invoice.pdf")
print(ocr.full_text)

pdf_tables = extract_tables_from_document("text-layer-invoice.pdf")
if pdf_tables:
    print(pdf_tables[0].to_csv())

Small in-memory RAG is also explicit:

from docintel_kit import index_documents, search_documents

index_documents(["report.pdf"], collection="demo")
hits = search_documents("What changed in revenue?", collection="demo", top_k=3)
for hit in hits.matches:
    print(hit.score, hit.chunk.text)

Keys, network use, and model downloads

• Importing the package, native parsing, spreadsheet ingestion, Tesseract OCR, and Camelot extraction do not require an API key.
• Calling a built-in Claude task without ANTHROPIC_API_KEY raises before an HTTP request is made. Set the variable only when you intentionally use those functions; never commit keys.
• extract_fields() lazily downloads impira/layoutlm-document-qa from Hugging Face on first use unless it is already cached.
• RAG lazily downloads sentence-transformers/all-MiniLM-L6-v2 on first embedding request unless cached.
• The default layout backend attempts to obtain PubLayNet artifacts, but the upstream links are currently unavailable. Use trusted local config/weights or register a different backend.

Consequently, the toolkit is local-execution-oriented but does not promise a no-network default for every capability. Cache model artifacts ahead of time for controlled or offline environments and verify their licenses and provenance.

Known limitations

• Layout outage: LayoutParser's built-in PubLayNet model-zoo URLs currently resolve to deleted Dropbox resources (upstream issue #168). analyze_layout() raises an actionable error. This also affects the scanned-table layout+OCR fallback and table-typed fields in extract_fields().
• First-page scalar extraction: extract_fields() rasterizes the document but currently supplies page index 0 to the scalar QA model. Fields that exist only on later pages will not be found.
• In-memory RAG: the default vector store is process-local, brute-force, and nonpersistent. Reuse the same embedding model at index and query time. It is intended for prototypes and small corpora.
• Scans are not automatically OCRed for RAG: index_documents() uses parse_document() text. Build an OCR-to-index path in your application for image-only content.
• Table fallback is heuristic: layout+OCR groups nearby words; it is not robust cell reconstruction. Camelot exceptions may result in fallback behavior.
• No legacy .xls: use .xlsx, .xlsm, CSV, or TSV.
• No automatic pipeline: each function is independently invoked; applications own routing, retries, caching, persistence, and quality thresholds.

Testing and release checks

The repository contains deterministic unit/format tests plus environment-heavy integration tests that use Tesseract, Camelot, LayoutLM, and sentence-transformers. Counts vary with installed binaries, model cache state, and the known layout outage, so this README does not freeze a pass count.

python -m pytest testcases -v
python -m build
python -m twine check dist/*

CI runs Python 3.11 deterministic tests and package build/twine checks. Some tests are excluded from CI because their outcome depends on the host environment rather than on this package: model-backed integration tests (extraction, layout, RAG), two OCR text-accuracy assertions, the Java-backed Tabula test, and the CJK assertion in test_unicode_multilingual_text_preserved (its PDF fixture needs a broad-coverage Unicode TTF, which GitHub runners do not provide; Noto CJK ships as OTF/OTC and cannot be embedded by the fixture generator). Run the full suite locally to exercise all of them.

License

MIT © Kanishk S. See LICENSE.