docpipe

Unified document parsing, structured extraction, vector ingestion, and RAG pipeline SDK.

Overview

docpipe connects document parsing (Docling, MarkItDown, GLM-OCR), LLM-based structured extraction (LangExtract + LangChain), vector ingestion (pgvector or optional vector-store plugins), and RAG querying into a single composable pipeline. Optional integrations include Qdrant and S3-compatible adapters, authenticated MCP, opt-in in-memory or Redis RAG response caching, and AutoGen agents.

Four pipelines, composable together:

1. Parse — Unstructured docs → parsed text/markdown
2. Extract — Text → structured entities via LLM
3. Ingest — Chunks → embeddings → your vector store
4. RAG — Questions → grounded answers with citations (six retrieval strategies)

docpipe does not own your RAG data — each client passes connection_string on /ingest. An optional control-plane DB (SQLite in Docker) stores admin login and opt-in audit metadata only.

Full documentation (install extras, Docker, API reference, RAG strategies, observability, vector stores, plugins): docpipe docs · Source documentation map · Marketing site

Install

pip install docpipe-sdk
# API server + OpenTelemetry (optional)
pip install "docpipe-sdk[server,observability]"

Optional extras (docling, openai, google, pgvector, turbovec, rag, rag-redis, mcp-server, rerank, http, all, …) are listed on the Install guide. See docs/MCP_SERVER.md for hosted MCP setup and docs/RAG_CACHE.md for shared RAG caching.

The source and vector-store plugin API is experimental; installed providers use a namespaced provider/options envelope while legacy request fields remain supported. Qdrant and S3-compatible adapters have conformance coverage. S3 integration tests run against SeaweedFS; MinIO-specific interoperability is not claimed. See the plugin architecture, configuration guide, and external plugin example.

For unreleased commits: pip install git+https://github.com/thesunnysinha/docpipe.git

Quick start

import docpipe

# Parse (docling default; markitdown for lightweight Office/PDF → Markdown)
doc = docpipe.parse("invoice.pdf", parser="markitdown")
print(doc.markdown)

# Extract
schema = docpipe.ExtractionSchema(
    description="Extract invoice line items with amounts",
    model_id="gemini-2.5-flash",
)
results = docpipe.extract(doc.text, schema)

# Ingest + RAG (configure your DB + providers)
config = docpipe.IngestionConfig(
    connection_string="postgresql://user:pass@localhost:5432/mydb",
    table_name="invoices",
    embedding_provider="openai",
    embedding_model="text-embedding-3-small",
)
docpipe.ingest("invoice.pdf", config=config)

rag_config = docpipe.RAGConfig(
    connection_string=config.connection_string,
    table_name=config.table_name,
    embedding_provider="openai",
    embedding_model="text-embedding-3-small",
    llm_provider="openai",
    llm_model="gpt-4o",
    strategy="hyde",
    system_prompt=(
        "Answer using ONLY the context below.\n\n"
        "Context:\n{context}\n\nQuestion: {question}\n\nAnswer:"
    ),
    hyde_prompt="Write a passage that answers: {question}",
)
result = docpipe.query("What is the total on the invoice?", config=rag_config)
print(result.answer)

# Optional: AutoGen agents with vector-search tools (pip install "docpipe-sdk[autogen]")
agent_result = docpipe.agent_query(
    "What is the total on the invoice?",
    config=rag_config,
    enable_reviewer=True,
)
print(agent_result.answer)

CLI: docpipe parse, docpipe ingest, docpipe rag query, docpipe plugins list, docpipe profiles list, docpipe serve — see CLI & API server.

Docker (profile tags):

docker pull ghcr.io/thesunnysinha/docpipe:balanced   # default production
docker pull ghcr.io/thesunnysinha/docpipe:slim       # lightweight
docker pull ghcr.io/thesunnysinha/docpipe:quality    # OCR + BGE rerank
docker pull ghcr.io/thesunnysinha/docpipe:agents     # AutoGen
docker pull ghcr.io/thesunnysinha/docpipe:mcp        # Streamable HTTP MCP server

pip profiles: profile-slim, profile-balanced, profile-quality, profile-agents, profile-mcp, and profile-gpu. profile-gpu is available for custom installs/builds; there is no published :gpu image. See .env.example and docs/INTEGRATION.md for the current image/profile matrix.

Runtime presets on /ingest and /rag/query: preset=fast|balanced|quality|agents. Discover options via GET /profiles and GET /plugins.

Shared Kubernetes API (one docpipe for Jingo, Andocs, and other apps): manifests in k8s/, deploy via .github/workflows/deploy-k8s.yml. Consumers call http://docpipe.docpipe.svc.cluster.local:8000 and pass their own connection_string on each /ingest and /rag/* request (vectors stay in each app's Postgres). See env/k8s/DOCPIPE_ENV.example.

Docker examples (compose stacks + full env flag reference): examples/ — start with examples/internal-shared/ for a shared internal instance.

Learn more

License

MIT — see LICENSE.