ingestlib

Self-hosted document intelligence for RAG pipelines. One library that takes a raw document — PDF, DOCX, PPTX, or a PNG/JPEG/WebP image — and produces searchable, cited, retrieval-ready chunks, plus schema-driven extraction with verified citations: the territory of LlamaParse / Reducto / Unstructured.io, running on your own stack.

from ingestlib.services import ingest, retrieve

ingest("finance-10k.pdf")            # parse → classify → split → embed → vector store
result = retrieve("what were the total revenues?")
print(result.context)                # ranked chunks, each citing doc · page · section

Documentation: langmodule.github.io/ingestlib — guides for every stage, the full configuration reference, and the API docs.

What it does

Engines: PaddleOCR-VL-1.6 (0.9B VLM, runs on your GPU) for layout + recognition, Amazon Nova 2 Lite for judgment (chart reading, review, classification, chunk boundaries), Nova multimodal embeddings, eight vector stores (Pinecone, Qdrant, SQLite, Postgres/pgvector, MongoDB, Milvus, OpenSearch, Weaviate — all hybrid dense + sparse), S3 or a local folder for artifacts (artifact_store: s3 | local). ~$0.002/page in LLM spend. An OpenAI backend (GPT-5 vision-capable chat + text-embedding-3) ships alongside Bedrock — flip llm_provider: openai / embedding_provider: openai to run the whole pipeline on it instead — or ollama to keep every LLM call on your own machine. See below.

Quickstart

1. Requirements

• Python 3.12+ and uv
• AWS account with Bedrock access (us-east-1): Nova 2 Lite + Nova 2 multimodal embeddings — the default provider; the OpenAI backend or a local Ollama can run the whole pipeline instead (see below)
• Vector database — none at all by default: the sqlite connector stores vectors in a local file. Or a Pinecone account (serverless, free tier works), a Qdrant server (local docker or Qdrant Cloud), a Postgres with pgvector (RDS/Supabase/Neon or self-hosted), a MongoDB with search (Atlas any tier or 8.2+ self-managed), a Milvus (local docker or Zilliz Cloud), an OpenSearch (Amazon domain or local docker), a Weaviate (local docker or Weaviate Cloud) — each just one connection URL
• Internal registry (Postgres) — ingestlib's own metadata store, required for the corpus path (ingest/retrieve/lifecycle). Bring it up with the bundled compose file: docker compose -f infra/docker-compose.yml --profile registry up -d, then ingestlib registry init. Standalone operations (parse/classify/split/ extract on a single document) need no registry
• Jina AI account for reranking (free tier: 100 RPM) — the default; or set reranker: aws (Amazon Rerank, same AWS credentials) or reranker: none in config.yaml and skip Jina entirely

2. Install

uv add ingestlib               # or: pip install ingestlib

The core install covers the full default stack (including sqlite vectors). Server-backed vector stores are extras — add the one you'll use:

uv add "ingestlib[qdrant]"     # pinecone · qdrant · pgvector · mongodb
                               # · milvus · opensearch · weaviate · all

Or work from source:

git clone https://github.com/LangModule/ingestlib.git
cd ingestlib
uv sync

System dependency — LibreOffice (DOCX/PPTX → PDF conversion):

brew install --cask libreoffice          # macOS (binary is `soffice`)
sudo apt install libreoffice-core libreoffice-writer libreoffice-impress   # Linux

Optional — PostgreSQL client tools (pg_dump/pg_restore), only for ingestlib registry backup/restore:

brew install libpq                       # macOS
sudo apt install postgresql-client       # Linux

3. Start the OCR inference server

Parse runs PaddleOCR-VL-1.6 behind an inference server. First launch downloads ~1.8 GB of weights; later launches load from cache in seconds.

# Apple Silicon (Metal GPU)
uv run python -m mlx_vlm.server --port 8111 --model PaddlePaddle/PaddleOCR-VL-1.6

# NVIDIA (then set paddle_vl.backend: vllm-server in config.yaml)
vllm serve PaddlePaddle/PaddleOCR-VL-1.6 --port 8111

The layout model (PP-DocLayoutV3, ~126 MB) auto-downloads on the first parse.

4. Configure

ingestlib init scaffolds the setup files into the current directory:

uv run ingestlib init            # default stack: Bedrock + sqlite + Jina → config.yaml + .env
uv run ingestlib init --local    # zero-cloud: Ollama + sqlite + local artifacts — no keys at all

For the default stack, add Bedrock-enabled credentials:

aws configure --profile your-aws-profile

Edit config.yaml: pick your providers, vector store, reranker, and artifact store — everything else has working defaults (vector_store: sqlite needs no server and no keys) — and fill .env with the keys your choices need (Jina for the default reranker; --local needs none). The aws section is required only while a choice uses AWS (the default bedrock provider, s3 artifacts, the aws reranker, an Amazon OpenSearch domain) — delete it otherwise and the config loader will tell you if something still needs it. The S3 bucket (default ingestlib-{account_id}) and the vector indexes/collections are created automatically on first use — no manual setup. Prefer no cloud storage at all? artifact_store: local keeps every source file, page render, and figure crop in a plain folder beside your config.yaml — browsable in a file manager, and moving the artifact store between backends is a plain copy.

Config is discovered at call time, never at import: INGESTLIB_CONFIG=/path/to/config.yaml wins, otherwise the working directory and its parents are searched — so installed usage works the same as running inside this repo.

5. Verify

uv run ingestlib doctor

Doctor checks every configured choice with real calls — config discovery, LibreOffice, the OCR server, an LLM ping, an embedding (with its dimension), the reranker, the artifact store, the vector store, and the registry (reachable and migrated). Every failed line prints the one-sentence fix (wrong AWS profile → your available profiles, missing key → where to get one, model not pulled → the exact ollama pull).

6. Run

from ingestlib.services import ingest, retrieve

r = ingest("report.pdf")
print(r.status, r.category, r.chunks, r.durations)

res = retrieve("what does the report conclude?", top_k=5)
for hit in res.hits:
    print(hit.rerank_score, hit.citation, hit.chunk.heading)

Manage a corpus

Real corpora change. Re-ingesting an edited file replaces the old version (its vectors and artifacts are deleted — no stale chunks), a rename is a cheap move, and sync() reconciles a whole folder in one call:

from ingestlib.services import ingest, sync, remove, reindex

ingest("report.pdf")                       # edited file → status="replaced"
sync("corpus/", prune=True)                # add new, replace changed, drop deleted
remove("old.pdf")                          # erase one doc from both stores
reindex()                                  # rebuild the vector store from the registry

The same verbs are on the CLI — a corpus is managed from the shell, no Python needed:

ingestlib ingest report.pdf docs/          # files or folders
ingestlib sync corpus/ --prune --dry-run   # preview, then drop --dry-run
ingestlib list                             # every stored document
ingestlib remove report.pdf                # erase one
ingestlib reindex                          # rebuild the index (provider switch, new store)
ingestlib recollect                        # re-sort the corpus after editing classify rules
ingestlib search "what were the risks?"    # cited retrieval from the shell

Documents carry a logical identity (namespace + source path), so lifecycle knows a re-ingest from a brand-new file. Full guide: Manage a corpus.

Query your databases

retrieve() can also answer from your SQL databases, alongside the document corpus, in the same call. Declare the databases in a sources.yaml sidecar (beside config.yaml, like rules.yaml — see sources.example.yaml) and pass their names:

from ingestlib.services import retrieve

# documents AND databases behind one question
result = retrieve("how many prescriptions are ready?", sources=["prescriptions"])
for r in result.results:
    print(r.source_type, r.content, r.provenance["sql"])   # the exact query that ran

The model generates read-only SQL from your schema + tables hints, bounded by a permission boundary whose floor is a read-only database role — so a wrong query is a wrong read, never a write. A statement allowlist, an injected LIMIT, and a timeout are defense in depth; verified queries let you pin reviewed SQL for answers that must be exact. On a wide schema ingestlib retrieves only the tables a question needs — plus their foreign-key bridges — instead of dumping every table into the prompt (schema_rag: auto). Generate hints for a cryptic schema with ingestlib describe-schema, and measure generation accuracy on your own schema with ingestlib eval-sql. Each SQL backend needs its pip extra (ingestlib[postgres] · [mysql] · [duckdb] · [snowflake]; sqlite needs none). From the shell: ingestlib search "…" --sources prescriptions. Full guide: Query databases (SQL).

Serve it to agents (MCP)

ingestlib mcp exposes the whole loop — search, extract, ingest, sync, remove, reindex, recollect, verify, and registry management — as MCP tools, so Claude Desktop, Cursor, or any agent can drive your self-hosted corpus with citations, nothing leaving your machine.

pip install "ingestlib[mcp]"
ingestlib mcp                                # stdio (local clients)
ingestlib mcp --transport http --port 8000   # remote; needs MCP_TOKEN (bearer auth)

Read tools (search/extract/classify/list/doctor) are always on; the write tools hide under --read-only. Guide: Serve to agents (MCP).

Using the operations directly

Every operation also works standalone:

from ingestlib.operations import parse, classify, split

result = parse("report.pdf")            # ParseResult: pages, regions, figures
print(result.markdown)                  # whole-document markdown
result.save_images("out/")              # extracted figures/charts as PNGs

label = classify("report.pdf")          # no OCR needed — native text + embedded images
chunks = split(result, category=label.category)
for c in chunks.chunks:
    print(c.token_estimate, c.embedding_text.splitlines()[0])

And the fourth operation pulls structured data out — your schema, filled and cited:

from pydantic import BaseModel
from ingestlib.operations import extract

class Receipt(BaseModel):
    merchant: str
    total: float
    currency: str

report = extract(parse("expenses.pdf"), schema=Receipt, mode="many")
for item in report.items:
    print(item.value.merchant, item.value.total, item.citation)
# BART 20.0 p.10  ·  Hilton 214.6 p.2  ·  …
print(report.items[0].fields["total"].grounded)   # True — verified in the cited region

mode="one" fills a single instance from the whole document (a 10-K's headline financials); mode="many" finds every instance across the pages. A ParseResult input gives region-level citations on scans; a raw path reads the native text layer with page-level citations and no OCR server. Confidence is honest: a field whose citation doesn't check out is capped, and a value not found in its cited text is flagged grounded=False.

A stored document reads back explicitly — queryable metadata from the registry, bytes from the blob store:

from ingestlib.services import get_document

doc = get_document(doc_id)              # the stored document, from the registry
print(doc.category, doc.chunk_count)    # queryable metadata
doc.markdown()                          # whole-document markdown (blob store)
[e["schema_name"] for e in doc.extractions]   # any persisted extractions

Classification & split rules

Classify and split are open-ended by default — the LLM decides the document type and discovers the section vocabulary. Both can instead follow your rules: pass them per call, or preset them once in rules.yaml beside your config.yaml, and every bare call and the whole ingest pipeline uses them automatically:

classify("doc.pdf",
         {"invoice": "Itemized charges, tax info, and payment terms",
          "sec_filing": "10-K/10-Q style regulatory filings"},
         target_pages="1,3,5-7", max_pages=5)   # read only these pages

split("report.pdf",
      vocabulary={"financial_statements": "Balance sheets, income statements",
                  "notes": "Footnotes and disclosures"},
      unmatched="other")   # pages fitting nothing: other (default) | require | skip

# rules.yaml — beside your config.yaml; infra stays in config.yaml,
# what your documents MEAN lives here
classify:
  max_pages: 5
  rules:                       # up to 20 — result is one of these or "uncategorized"
    invoice: "Itemized charges, tax info, and payment terms"
    sec_filing: "10-K/10-Q style regulatory filings"
split:
  unmatched: other             # require | other | skip
  categories:                  # up to 50 — YOUR sections; Pass 1 is skipped
    financial_statements: "Balance sheets, income statements, cash flows"
    notes: "Footnotes and disclosures"

Classify returns one of your labels or "uncategorized", with confidence, reasoning, and ranked alternatives. Split labels every page against your sections — unmatched pages become an honest other section (default), get forced into the nearest category (require), or are dropped entirely (skip). Precedence everywhere: explicit arguments beat the preset, and {} forces the open-ended default even when a preset exists.

OpenAI backend

The same LLM surface Bedrock provides is also available on OpenAI — GPT-5 chat with vision, thinking mode, schema-enforced structured output, and text-embedding-3 embeddings. Add OPENAI_API_KEY to .env and pick models in config.yaml's openai: section (defaults: gpt-5-mini, text-embedding-3-small).

To run the whole ingest/retrieve pipeline on it, switch the providers in config.yaml — every LLM and embedding call routes accordingly:

llm_provider: openai          # chart reading, review, classify, chunking
embedding_provider: openai    # chunk + query embeddings

Combined with artifact_store: local, vector_store: sqlite, and reranker: jina (or none), the pipeline needs no AWS at all. Two rules: switching embedding_provider changes the vector space, so re-ingest afterward (skip_existing=False) — vectors from different embedding models never mix in one index. And text embeddings only: OpenAI has no image-embedding model.

The backend is also importable directly, ignoring the config switch:

from ingestlib.foundations.llm import Image
from ingestlib.foundations.llm.openai import chat, chat_structured, embed_text

chat("Read this chart", images=[Image(png_bytes, "png")])   # vision works
embed_text("a chunk of text")                               # 1024-dim default

Local backend (Ollama)

The third provider keeps every LLM call on your machine — the fully air-gapped pipeline. Point config.yaml at a local Ollama (or any OpenAI-compatible server: vLLM, LM Studio):

llm_provider: ollama
embedding_provider: ollama

ollama:                                    # these are the defaults
  base_url: http://localhost:11434/v1
  llm_model_id: qwen3.5:9b
  embedding_model_id: qwen3-embedding:0.6b

ollama pull qwen3.5:9b
ollama pull qwen3-embedding:0.6b

No API key. Vision, schema-enforced structured output, and 1024-dim embeddings all verified on the reference stack — and local embedding quality is measured, not assumed: on the retrieval eval, qwen3-embedding matched or beat the cloud stack (perfect reranked hit@1/3/5 = 1.00). Two honest notes: use the GGUF builds, not -mlx (Ollama's MLX engine silently drops images and schema enforcement), and a local 9B won't match the cloud models on dense charts — test with your own documents before committing. Combined with artifact_store: local and vector_store: sqlite, nothing leaves your machine but the optional Jina rerank call (reranker: none closes even that).

Architecture

src/ingestlib/
├── services/       ingest · retrieve · lifecycle (remove · sync · reindex · recollect) · verify — the product
├── operations/     parse · classify · split · extract — the tools (each standalone)
├── storage/        registry (Postgres metadata hub) · artifacts (S3 | local, bytes only) · base (VectorStore contract) · 8 connectors
│                   (pinecone · qdrant · sqlite · pgvector · mongodb · milvus
│                    · opensearch · weaviate)
├── sources/        structured retrieval — SQL databases & the corpus as queryable Sources
├── foundations/    llm (Bedrock Nova · OpenAI GPT-5 · Ollama Qwen · Jina) · ocr (PaddleOCR-VL)
├── cli/            the `ingestlib` command — init · doctor · ingest · sync · list · show · collections · remove · reindex · recollect · verify · search · describe-schema · eval-sql · registry · mcp
├── mcp/            MCP server (ingestlib[mcp]) — expose the pipeline to agents
├── utils/          logger · files · sync · aws
└── config.py       config.yaml + .env + rules.yaml + sources.yaml → typed configs

src/ingestlib_registry/   the registry's Postgres schema + Alembic migrations (a standalone package ingestlib depends on)

Strict downward dependencies. The VectorStore contract means backends drop in as connectors — all eight ship hybrid search: Pinecone (dense + hosted sparse model, merged client-side), Qdrant (dense + BM25 with server-side IDF and RRF fusion; local docker or cloud), SQLite (sqlite-vec KNN + built-in FTS5 BM25 with porter stemming, RRF fusion — one local file, no server, no keys), Postgres/pgvector (HNSW cosine + built-in full-text over a generated weighted tsvector, RRF fusion — the extension and table bootstrap automatically), MongoDB (Atlas Vector Search + Atlas Search true BM25, RRF fusion — Atlas any tier or self-managed 8.2+; both search indexes bootstrap automatically), Milvus (dense ANN + server-computed BM25 sparse, fused server-side with RRF — local docker or Zilliz Cloud), OpenSearch (faiss HNSW k-NN + Lucene BM25, RRF fused client-side — an Amazon OpenSearch domain SigV4-signed with your aws profile, or local docker), and Weaviate (HNSW dense + native BM25 fused server-side in one hybrid call — local docker or Weaviate Cloud). Pick one with vector_store: pinecone | qdrant | sqlite | pgvector | mongodb | milvus | opensearch | weaviate in config.yaml. Connection secrets sit in .env together (sqlite needs none) — only the selected connector ever builds a client.

Logging

INGESTLIB_LOG_LEVEL=INFO           # DEBUG | INFO | WARNING | ERROR (default INFO)
INGESTLIB_LOG_THIRD_PARTY=1        # also show paddlex/httpx/botocore chatter
INGESTLIB_LOG_COLOR=0              # disable colored output

Testing

Tests hit real APIs, never mocks. Pure logic runs always; server-hitting suites are opt-in via env gates. The sqlite connector's full suite runs ungated in make test — there is no server, so in-process IS the real thing.

make test                  # fast suite (~630 tests, ~2min; e2e groups skip)
make test-openai           # OpenAI backend       (skips without OPENAI_API_KEY)
make test-ollama           # Ollama backend       (needs a local Ollama + models)
make test-parse            # parse e2e            (needs VL server + LLM provider)
make test-classify         # classify e2e         (needs the LLM provider)
make test-split            # split e2e            (needs the LLM provider)
make test-extract          # extract e2e          (needs the LLM provider; scans need the VL server)
make test-s3               # artifact store e2e   (needs AWS)
make test-pinecone         # vector connector e2e (needs Pinecone + embeddings)
make test-qdrant           # vector connector e2e (needs a Qdrant server + embeddings)
make test-sqlite           # vector connector suite (no gate — nothing to need)
make test-pgvector         # vector connector e2e (needs Postgres at PGVECTOR_URL)
make test-mongodb          # vector connector e2e (needs MongoDB at MONGODB_URL)
make test-milvus           # vector connector e2e (needs Milvus at MILVUS_URL)
make test-opensearch       # vector connector e2e (needs OpenSearch at OPENSEARCH_URL)
make test-weaviate         # vector connector e2e (needs Weaviate at WEAVIATE_URL)
make test-services         # full product e2e     (needs the entire stack)
make test-sources          # structured retrieval — SQL sources (deterministic; e2e gated)
make test-cli              # CLI: init/doctor + corpus commands (no gate)
make test-lifecycle        # remove/sync/reindex/recollect + replace-aware ingest (gates on the registry)
make test-mcp              # MCP server: tools, read_only, http auth (no gate)
make test-all              # everything
make eval                  # retrieval quality eval (see below)
make docs                  # live-preview the documentation site

Fixtures live in tests/data/ — 15 real PDFs (research papers, earnings decks, insurance forms, 10-Ks, a 16-page receipt scan, a password-protected one), document images incl. WebP, and real DOCX/PPTX files. The server-backed connector suites need their servers: docker compose -f infra/docker-compose.yml --profile all up -d starts every local store (or --profile qdrant etc. for just one).

Retrieval quality

Beyond pass/fail tests, evals/ measures retrieval quality: 22 ground-truth questions over the fixture corpus, run through the real retrieve() flow under dense/hybrid × rerank on/off, scored by hit@k and MRR. Measured so far (consistent across all eight connectors): with reranking, every answer lands in the top 3 hits (hit@3 = 1.00); hit@1 ranges 0.86–1.00 across runs. The fully-local stack holds the same bar — Ollama embeddings scored a perfect 1.00 across hit@1/3/5 and MRR on dense+rerank. Each run saves a timestamped snapshot to evals/results/, so quality changes are visible over time.

Disk footprint

Scope

English documents; PDF / DOCX / PPTX / PNG / JPEG / WebP input. Images, charts, and tables inside documents are fully extracted and interpreted, and a single image file parses as a one-page document; handwriting is out of scope by design.

The studio

ingestlib-studio is the visual companion: a local web UI with a setup wizard, try-before-you-commit pipeline runs, page-by-page review with hover-to-highlight bounding boxes, committed ingestion with live progress, a content-rules editor, and a retrieval playground where every answer points to its source on the page.

Roadmap

• XLSX input (tables-first, not a PDF conversion)

Recently shipped: the internal registry — a Postgres metadata hub that makes the whole corpus queryable, with reindex/recollect/verify for rebuilding and auditing it and event-driven backups (v1.5); schema-RAG for wide databases — retrieve the relevant tables (with foreign-key closure) instead of dumping the whole schema, plus describe-schema auto-documentation and the eval-sql accuracy harness (v1.4); structured retrieval — query your SQL databases alongside documents through one retrieve() call, behind a read-only permission boundary (v1.3); an MCP server to serve the corpus to agents (v1.2); document lifecycle — replace-aware ingestion, folder sync(), reindex(), and the corpus CLI (v1.1).

License

See LICENSE.