Skip to main content

docpipe-maestro

PyPI CI Python 3.10+ License: MIT Code Style: Ruff Type Checked: mypy

"docpipe-maestro is a minimal RAG pipeline: it ingests any document, chunks it correctly, embeds it, stores it, retrieves the right sections, and optionally generates a grounded answer. It is the engine — not the app, not the framework."

docpipe-maestro is a provider-agnostic, structure-aware Python RAG (Retrieval-Augmented Generation) pipeline library and docpipe-maestro CLI. It solves the data-preparation and retrieval failure modes common in standard fixed-size chunkers: preserving document hierarchy, refusing to split legal or policy clauses mid-sentence, resolving intra-document cross-references, and guaranteeing that policy exceptions always stay attached to their rules.


Table of Contents


Why docpipe-maestro?

Common RAG Failure Mode How docpipe-maestro Solves It
Orphaned exceptions ("20 days leave" retrieved without "except probationary hires") Parent-child retrieval + POLICY clause preservation searches fine-grained child chunks and returns deduplicated full parent sections, enforced by the mandatory Reverse Safety Gate (tests/test_reverse_safety.py).
Broken intra-document references ("subject to Section 4.2" without knowing what Section 4.2 says) POLICY CrossrefsEnricher (enrichers=("crossrefs",)) appends cycle-safe [Crossref ...] excerpts directly onto referring sections during Phase 1 ingestion (toggleable at query time via --no-crossrefs / follow_crossrefs=False).
Exact identifiers & ranking misses in dense-only search (SEC-101, IR-15M) Hybrid search + child-first reranking pairs collocated SQLite FTS5 BM25 + dense vector RRF (fts5_hybrid) with optional local cross-encoder reranking (--rerank / rerank=True).
Cross-corpus noise across files, sections, profiles, or dates Unified where metadata & manifest filtering compiles validated filter operators ($eq, $ne, $in, $nin, $gt, $gte, $lt, $lte, $and, $or) over chunk fields and manifest attributes (profile, ingested_after, ingested_before) with ChromaDB and FTS5 parity and zero re-ingestion.
Orphaned vectors when files shrink or change Transactional doc_version staging writes new chunks, flips the JSON manifest atomically, and deletes stale versions afterward (with startup sweep_uncommitted() recovery). Failed runs leave the prior version intact.
Silent embedding vector-space corruption Manifest model lock pins each collection to one embed_model and raises ModelMismatchError unless migrated explicitly via reindex.
Hallucinations when nothing relevant matches Deterministic empty-context refusal returns "Not found in the provided context." without calling the LLM (ask checks retrieved hits automatically; answer()/answer_detailed() short-circuit when has_context=False).
Sensitive PII stored or sent to remote LLMs Opt-in pre-chunking PiiEnricher + outbound guard_egress() redact, HMAC-tokenize, or block entities cleanly (PiiBlockedError), log zero-raw-PII .audit.jsonl records (0o700/0o600), and enforce --local-only pre-network across embeddings and generation.
Hidden HTML/XLSX/EPUB injections & unsafe archives Hardened intake strips hidden DOM/EPUB elements (display:none, hidden, aria-hidden, <template>), skips hidden/veryHidden Excel sheets, disables RST file-inclusion/raw directives, enforces single-pass ZIP uncompressed size caps (256 MiB) across .docx/.xlsx/.pptx/.epub, rejects legacy .doc/.xls/.ppt binaries, and skips directory symlinks.
Blind retrieval regressions, vendor lock-in & library bloat Offline eval harness, Strategy registries & Library Purity deliver reproducible MRR / Precision@k / Recall@k checks, 9 @register_* extension points, NullHandler-only docpipe_maestro.log (--log-json stderr telemetry), zero config-file reads in library code, and LiteLLM routing with zero hardcoded provider registry.

60-Second Quickstart

Run an entirely local pipeline using Ollama (nomic-embed-text + llama3.2):

# 1. Install from PyPI with all format parsers
pip install "docpipe-maestro[all]"

# 2. Pull local Ollama models (one-time)
ollama pull nomic-embed-text
ollama pull llama3.2

# 3. Ingest a folder of policies and ask a grounded question
docpipe-maestro ingest ./policies --profile POLICY --collection handbook
docpipe-maestro ask "When are employees not entitled to overtime?" --collection handbook --threshold 0.35

(Running from a cloned source repository? Prefix CLI commands with uv run docpipe-maestro ... or uv run python -m docpipe_maestro ....)


Key Architecture & Core Concepts

flowchart LR
    subgraph P1["Phase 1: Offline Ingestion (Atomic & Transactional)"]
        A[Source Files] --> B[Intake & Format Detect]
        B --> C[Clean & Boilerplate Strip]
        C --> D["Enrichers (PII -> Crossrefs)"]
        D --> E[Structure-Aware Chunker]
        E --> F["local_only Check + Batched Embeddings"]
        F --> G[(ChromaDB + FTS5 + Manifest + .audit.jsonl)]
    end

    subgraph P2["Phase 2: Online Query (Read-Only)"]
        Q[Question] --> QE["local_only Check + Query Embed"]
        QE --> S["Child Search (Dense or fts5_hybrid RRF)"]
        S --> RR["Optional Rerank (Cross-Encoder Sigmoid)"]
        RR --> PE["Parent Expansion & max(score) Dedupe"]
        PE --> BP["Pure build_prompt() / build_chat_messages()"]
        BP -->|results > 0| EG["Egress Guard (local_only + PII Policy)"]
        EG --> LLM["Optional answer() via LiteLLM"]
        BP -->|empty results| REF["Deterministic Refusal (No LLM Call)"]
    end
  1. Two-Phase Separation:
    • Phase 1 (Offline Ingestion): Files $\to$ Parse $\to$ Clean $\to$ Enrich (pii $\to$ crossrefs) $\to$ Structure-Aware Chunking $\to$ Embeddings $\to$ Vector Database + .audit.jsonl. Correctness and crash safety matter.
    • Phase 2 (Online Query): Question $\to$ Embed $\to$ Retrieve (Match Children, Return Deduplicated Parents) $\to$ Build Prompt $\to$ Egress Guard $\to$ Optional Grounded LLM Generation. Read-only and low-latency.
  2. Search Small, Return Big (Parent-Child Retrieval):
    • Small child chunks (individual clauses/paragraphs prefixed with [Source: ... | Section: ...]) are embedded and searched for high precision.
    • Large parent sections (complete contextual sections) are stored with zero-vector placeholders and returned to the prompt.
  3. Exception Preservation & Reverse Safety Gate:
    • In policy and legal documents, a rule without its exception is actively misleading (e.g., "Employees receive 20 days leave" without "unless probationary"). In POLICY profile, clauses are kept intact even when exceeding target chunk size, and parent sections ensure exceptions always accompany their rules.
  4. Crash-Safe Transactional Synchronization:
    • Ingestion maintains an atomic database-collocated manifest (.docpipe/manifests/<collection>.json) tracking sha256 content hashes, mtime, and doc_version prefixes.
    • Modifying a document writes new versioned chunks, commits the manifest atomically, and deletes stale versions afterward, preventing orphaned clauses when documents shrink.
  5. Embedding Model Lock:
    • The manifest locks the collection to the embedding model used during ingestion. Attempting to ingest or query with a different model raises ModelMismatchError rather than silently mixing incompatible vector spaces. Model migration is handled cleanly via reindex.

Supported Formats & Limitations

Format Extensions Structural & Content Extraction Extra Required
Markdown .md, .markdown ATX headings (#–######), code fence awareness (``` / ~~~ ignores # comments) Core install
Plain Text .txt Heuristic Section / Article / Chapter / Part and numbered headings Core install
CSV / TSV .csv, .tsv Strict UTF-8 / BOM normalization, trimmed empty trailing columns, ragged-row tolerance, 25-row bounded Header: Value sections Core install
HTML .html, .htm <h1–h6> headings, paragraphs, lists, tables (|), strips <script>/<style>/<nav>/<footer>/<noscript> & hidden subtrees [html] (beautifulsoup4)
Word .docx Word Heading 1–Heading 6 styles, paragraphs, tables (|) [docx] (python-docx)
PDF .pdf Per-page text, 1-based page numbers, multi-page active heading carryover, boilerplate header/footer stripping [pdf] (pypdf)
Excel .xlsx Visible sheets only (skips hidden/veryHidden), evaluated cached values, 25-row bounded Header: Value sections [xlsx] (openpyxl)
PowerPoint .pptx Per-slide titles & 1-based page numbers, body text, grouped shapes, tables (|), speaker notes [pptx] (python-pptx)
EPUB .epub Ordered spine chapters with <h1–h6> section headings, offline hidden-DOM/boilerplate stripping [epub] (ebooklib)
reStructuredText .rst Section/heading hierarchy (heading_level), paragraphs, lists, tables (|), admonitions; local file/raw directives disabled [rst] (docutils)
  • Extensionless files: detect_format() inspects the first 4 KiB for %PDF-, legacy OLE2 binaries (rejected cleanly), single-pass PK\x03\x04 ZIP containers (word/document.xml → docx, xl/workbook.xml → xlsx, ppt/presentation.xml → pptx, META-INF/container.xml → epub), HTML root tags (<!doctype html / <html), Markdown headings, or valid UTF-8 text.
  • Current scope & limitations:
    • Image-only / scanned PDFs without an embedded text layer yield empty text (OCR is out of scope).
    • Legacy binary .doc, .xls, and .ppt files are not supported (modern .docx, .xlsx, .pptx, or .csv/.tsv only).
    • docpipe-maestro is a library and CLI engine — it deliberately excludes web servers, auth layers, streaming UIs, and multi-tenant isolation.

Installation Guide

docpipe-maestro requires Python 3.10+.

Core Install

The core installation includes Markdown (.md, .markdown), plain text (.txt), CSV (.csv), and TSV (.tsv) parsing, embedding generation, ChromaDB vector storage, and semantic retrieval:

# Install from PyPI
pip install docpipe-maestro

# Or install the latest development head from GitHub:
# pip install "git+https://github.com/ivnmoreno/docpipe-maestro.git"

Optional Extras

Generation (answer(), ask) is available in the base install — no extra needed. Install optional extras for additional document format parsers:

Extra Purpose Dependencies Installed
pdf Extract text, page numbers, and heuristic section headings (Section X.Y) from PDF files pypdf>=6.21.3 (BSD-3-Clause)
docx Extract text, style-based headings (Heading 1–Heading 6), and tables from Word documents python-docx>=1.1.0
html Strip boilerplate and parse <h1–h6> headings from web pages / HTML exports beautifulsoup4>=4.12.0
xlsx Extract visible Excel worksheets into header-paired 25-row sections openpyxl>=3.1.0
pptx Extract numbered slides, body/tables, grouped shapes, and speaker notes from PowerPoint decks python-pptx>=1.0.0
epub Extract ordered spine chapters and headings from EPUB books safely offline ebooklib>=0.18
rst Extract structured sections and tables from reStructuredText offline (.. include:: disabled) docutils>=0.20
all Installs all optional document format parsers (pdf, docx, html, xlsx, pptx, epub, rst) pypdf, python-docx, beautifulsoup4, openpyxl, python-pptx, ebooklib, docutils
rerank Second-stage local neural cross-encoder reranking (BAAI/bge-reranker-base default) sentence-transformers>=3.0.0
pii Optional multi-language NER PII recognition alongside deterministic recognizers presidio-analyzer>=2.2.0, presidio-anonymizer>=2.2.0

Note on Structure Extraction: Markdown, plain text, CSV/TSV, HTML, DOCX, PDF, XLSX, PPTX, EPUB, and RST extract headings, clauses, tables/rows, and slide/page metadata cleanly without external SaaS parsers. Note that [all] installs document parsers only; include [rerank] and/or [pii] alongside it ([all,rerank,pii]) when enabling reranking or Presidio NER. For offline [pii] usage, pre-install the required local spaCy models (e.g. en_core_web_sm, es_core_news_sm).

Install specific extras:

# PDF support only
pip install "docpipe-maestro[pdf]"

# PDF, DOCX, and XLSX support
pip install "docpipe-maestro[pdf,docx,xlsx]"

# Full installation with all document format parsers
pip install "docpipe-maestro[all]"

# Parsers + local cross-encoder reranking + optional NER PII detection
pip install "docpipe-maestro[all,rerank,pii]"

Development Setup

To contribute or run tests locally (uses PEP 735 [dependency-groups]):

# Clone the repository
git clone https://github.com/ivnmoreno/docpipe-maestro.git
cd docpipe-maestro

# Option A: Using uv (recommended)
uv sync --all-extras --group dev

# Option B: Using pip 24.1+ (PEP 735 dependency groups)
pip install --group dev -e ".[all]"
pre-commit install --hook-type commit-msg

Note on Versioning & Commitizen: Package versions are derived dynamically from Git tags via hatch-vcs. Use uv run cz check (to validate Conventional Commits) and uv run cz changelog (to update CHANGELOG.md), rather than cz bump.


Configuration Guide

docpipe-maestro enforces strict Library Purity: no library module reads external configuration files. Configuration is passed explicitly via function arguments or provider environment variables. Only the docpipe-maestro CLI optionally reads a --config TOML file.

1. Local Execution (Default: Zero Cloud Accounts Required)

Out of the box, docpipe-maestro defaults to local execution with Ollama:

  • Default Embeddings Model: ollama/nomic-embed-text
  • Default Generation Model: ollama/llama3.2

Make sure Ollama is installed and running (http://localhost:11434), then pull the default models:

ollama pull nomic-embed-text
ollama pull llama3.2

Tip — Choosing a local generation model: For multi-section policy synthesis (especially on CPU or limited RAM), ollama/qwen2.5:1.5b-instruct or ollama/qwen2.5:3b follows retrieved context more reliably than ollama/llama3.2 (which can over-refuse paraphrased queries) and <1B models (which are too small for grounded synthesis). Override the default with --model ollama/qwen2.5:1.5b-instruct or set model = "ollama/qwen2.5:1.5b-instruct" in docpipe.toml.

Air-gapped / strict zero-egress environments: By default, litellm attempts to download its public model-cost JSON map from GitHub when first imported before falling back to its bundled copy. Set export LITELLM_LOCAL_MODEL_COST_MAP=true in your shell to prevent any startup HTTP call on import.

2. Cloud Provider API Keys (.env or Shell Environment)

Copy the tracked template .env.example to .env (git-ignored) and restrict permissions:

cp .env.example .env && chmod 600 .env

Or export provider environment variables directly in your shell (shell variables always take precedence over .env):

# Groq serverless inference
export GROQ_API_KEY="your-groq-api-key"

# Together.ai serverless inference (TOGETHER_API_KEY alias is also recognized by LiteLLM)
export TOGETHERAI_API_KEY="your-together-api-key"

# OpenAI / Anthropic / Google Gemini
export OPENAI_API_KEY="your-openai-api-key"
export ANTHROPIC_API_KEY="your-anthropic-api-key"
export GEMINI_API_KEY="your-gemini-api-key"

# Air-gapped / offline startup cost-map safety
export LITELLM_LOCAL_MODEL_COST_MAP="true"

You can then pass any provider/model string directly to --embed-model or --model (e.g., groq/llama-3.3-70b-versatile, together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo, openai/gpt-4o-mini, gemini/gemini-2.5-flash). Note that the CLI loads .env strictly from the --config directory or current working directory (never walking parent directories), while direct Python API callers configure os.environ.

3. Local Persistence Storage Directory

By default, data is stored in ./.docpipe:

.docpipe/
├── chroma.sqlite3                    # ChromaDB embedded vector database
├── <uuid>/                           # ChromaDB vector segments and HNSW index files
├── lexical/
│   └── <collection>.db               # Collocated SQLite FTS5 BM25 lexical index (`fts5_hybrid`)
└── manifests/
    ├── <collection>.json             # Atomic JSON manifest tracking file hashes, versions, and locked model
    └── <collection>.audit.jsonl      # Append-only zero-raw-PII audit log (written when PII policy is active)

Customize the persistence directory with --persist-dir on the CLI or persist_dir= in Python. Never commit .docpipe/ to version control if your corpus is private.

4. CLI Configuration File (docpipe.toml)

When --config is omitted (and --no-config is not passed), the CLI automatically loads ./docpipe.toml from the current working directory if present (non-recursively; never walking parent directories). Create an optional TOML configuration file to set default CLI options (never store API keys or secrets here — explicit CLI flags override --config / ./docpipe.toml, which override built-in defaults):

# docpipe.toml
collection = "my_kb"
persist_dir = "./data/docpipe_index"
embed_model = "ollama/nomic-embed-text"
model = "ollama/llama3.2"
profile = "POLICY"
top_k = 5
threshold = 0.4
searcher = "default"               # "default" (dense vector) or "fts5_hybrid" (dense + SQLite FTS5 RRF)
# Optional second-stage cross-encoder reranking (requires `[rerank]` extra):
# rerank = false
# reranker = "cross_encoder"
# rerank_top_n = 20
# Optional online/gateway defaults for `docpipe-maestro ask` (CLI flags override these):
# api_base = "https://api.groq.com/openai/v1"
# temperature = 0.0
# max_tokens = 512                 # Keep omitted (or >= 4096) for reasoning models
# timeout = 60.0
# fail_on_truncation = false
# Privacy & PII defaults (JSON/TOML path string or `[pii_policy]` table applied in CLI only):
# local_only = false
# pii_policy = "./pii-policy.json"

Pass an explicit config path with --config, or ignore ./docpipe.toml with --no-config:

docpipe-maestro query "What is the probation period?" --config docpipe.toml
docpipe-maestro ask "What is the probation period?" --no-config

Optional PII Policy File (--pii-policy)

When enabling --pii-policy ./pii-policy.json (or .toml) on ingest, sync, or ask, configure actions ("redact", "tokenize", "block") per entity type (EMAIL_ADDRESS, IBAN_CODE, CREDIT_CARD, US_SSN, IP_ADDRESS, PHONE_NUMBER, ES_DNI, plus optional [pii] Presidio ner: true):

{
  "types": {
    "EMAIL_ADDRESS": "tokenize",
    "IBAN_CODE": "redact",
    "US_SSN": "block"
  },
  "token_salt_env": "DOCPIPE_PII_SALT",
  "egress": "redact"
}

(In docpipe.toml, set pii_policy = "./pii-policy.json" or define an inline [pii_policy] table. Use token_salt_env outside the repository for deterministic cross-run tokens.)

5. Chunking Profiles (DEFAULT, POLICY, and Custom Profiles)

Built-in profiles are immutable presets (docpipe_maestro.config) calibrated for specific document types:

Parameter DEFAULT Profile POLICY Profile
Intended For Notes, general prose, articles, documentation Handbooks, compliance policies, HR manuals, contracts
Target Chunk Size ~512 tokens ~1000 tokens
Chunk Overlap ~64 tokens (scoped within section) ~80 tokens (scoped within section)
Clause Preservation Disabled (splits on sentences/paragraphs) Enabled (never splits inside detected numbered clauses; PDF clause detection is heuristic)
Cross-Reference Resolution Disabled Enabled (resolves §2.1, Section 4, Clause 3.2)
Enrichers (enrichers) () (skipped) ("crossrefs",)
Parent-Child Expansion Enabled Enabled

Note on PDF Clause Detection: Markdown, DOCX, and HTML headings/numbering provide explicit structural boundaries, whereas PDF clause detection uses extracted-text heuristics (1.1, Clause 2, (a) layout cues) and depends on clean PDF text extraction order.

In Python, you can also pass a custom ChunkingProfile instance directly to ingest(..., profile=custom_profile) or sync(...), and configure ordered post-clean enrichers via profile.enrichers and @register_enricher.


Quickstart: CLI Usage Guide

Run docpipe-maestro --help to see all available subcommands (global flags such as --config, --no-config, --collection, --persist-dir, --local-only, --log-json, and --log-level work either before or after any subcommand):

usage: docpipe-maestro [-h] [--version] [--config CONFIG] [--no-config]
                       [--collection COLLECTION] [--persist-dir PERSIST_DIR]
                       [--local-only | --no-local-only]
                       [--pii-policy PII_POLICY] [--log-json]
                       [--log-level {DEBUG,INFO,WARNING,ERROR}]
                       {ingest,sync,status,remove,reindex,query,ask,eval} ...

See the full CLI Reference for all flags, TOML keys, exit codes, and output schemas.

1. Ingest Documents (docpipe-maestro ingest)

Ingest a single file or an entire directory tree atomically (if any file fails validation, no records are committed):

# Ingest with default profile
docpipe-maestro ingest ./docs --collection knowledge_base

# Ingest policy documents with clause-preservation and cross-reference resolution
docpipe-maestro ingest ./company_policies --profile POLICY --collection hr_policy

# Ingest strictly locally with pre-chunking PII scrubbing/blocking and .audit.jsonl output
docpipe-maestro ingest ./company_policies --local-only --pii-policy ./pii-policy.json --collection hr_policy

# Force re-ingestion even if files have not changed on disk
docpipe-maestro ingest ./docs --force --collection knowledge_base

# Ingest policies and export a self-contained offline HTML integrity & chunk audit report
docpipe-maestro ingest ./company_policies --profile POLICY --collection hr_policy --html-report ./audit.html

2. Keep Collections in Sync (docpipe-maestro sync)

Reconcile a target directory against the manifest (adds new files, skips unchanged files via mtime/sha256, updates modified files, and purges deleted files), or preview changes and offline token/cost estimates safely first with --dry-run (plus --json for CI gating):

# Preview deltas, chunks, tokens, and estimated cost without mutating disk or calling embedders
docpipe-maestro sync ./company_policies --profile POLICY --collection hr_policy --dry-run

# Apply reconciliation
docpipe-maestro sync ./company_policies --profile POLICY --collection hr_policy

Example output:

Synced 'hr_policy': added=1, updated=2, skipped=10, removed=1.

3. Inspect Collection Status (docpipe-maestro status)

View indexed files, active doc_version hashes, chunk counts, and locked embedding model:

docpipe-maestro status --collection hr_policy

Example output:

Collection: hr_policy | Model: ollama/nomic-embed-text (dim=768) | Sources: 2 | Children: 18 | Parents: 6
  - hr_leave_policy.md [4f1a29b01c3d] (10 children, 3 parents)
  - travel_expenses_policy.md [a8c2194ef001] (8 children, 3 parents)

Use --json for machine-readable output or --html-report to export a self-contained single-file HTML Document Integrity & Chunk Audit Report:

docpipe-maestro status --collection hr_policy --json
docpipe-maestro status --collection hr_policy --html-report ./audit.html

4. Remove a Source Document (docpipe-maestro remove)

Purge all child chunks, parent sections, and manifest entries for a specific file:

docpipe-maestro remove hr_leave_policy.md --collection hr_policy

5. Migrate Embedding Models (docpipe-maestro reindex)

Re-embed all existing child chunks stored in the database under a new embedding model, update parent vector dimensions, and re-lock the collection (or pass --dry-run [--json] to estimate active child chunks, tokens, and offline cost before making API calls):

# Preview active chunks and estimated embedding cost before migrating
docpipe-maestro reindex --model openai/text-embedding-3-small --collection hr_policy --dry-run

# Migrate collection to OpenAI embeddings
export OPENAI_API_KEY="sk-..."
docpipe-maestro reindex --model openai/text-embedding-3-small --collection hr_policy

6. Semantic Search Without an LLM (docpipe-maestro query)

Perform Phase 2 semantic retrieval to inspect scored passages and provenance without invoking a language model:

# Basic semantic query
docpipe-maestro query "How many vacation days do full-time employees accrue?" --collection hr_policy

# Restrict to top 3 sections with a minimum similarity threshold (0.0 to 1.0)
docpipe-maestro query "What is the flight reimbursement policy?" --top-k 3 --threshold 0.5 --collection hr_policy

# Combine dense vector similarity with SQLite FTS5 BM25 via Reciprocal Rank Fusion
# (ideal for exact error codes, identifiers, or clauses like "SEV-1" or "AES-256")
docpipe-maestro query "SEV-1 escalation window" --searcher fts5_hybrid --collection hr_policy

# Apply optional second-stage cross-encoder reranking over top candidate child chunks
# (requires `docpipe-maestro[rerank]`)
docpipe-maestro query "SEV-1 escalation window" --searcher fts5_hybrid --rerank --rerank-top-n 20 --collection hr_policy

# Return raw child chunks instead of expanded parent sections
docpipe-maestro query "annual leave" --no-expand-parent --collection hr_policy

# Scope search by document metadata or manifest profile/ingestion window (without re-ingestion)
docpipe-maestro query "annual leave" --where profile=POLICY --where page=2 --collection hr_policy
docpipe-maestro query "annual leave" --where ingested_after=2026-01-01 --collection hr_policy

# Emit structured JSON (for evaluation fixtures and scripts) or add --log-json telemetry on stderr
docpipe-maestro query "annual leave" --json --collection hr_policy
docpipe-maestro --log-json --log-level INFO query "annual leave" --collection hr_policy 2>trace.jsonl

Example output:

[1] score=0.8421 | Source: hr_leave_policy.md | Section: 3. Annual Leave
3.1 Full-time employees accrue 20 days of paid annual leave per calendar year.
3.2 Employees with less than one year of tenure accrue leave on a pro-rata basis.
------------------------------------------------------------

7. Grounded Question Answering (docpipe-maestro ask)

Retrieve relevant context, build a grounded prompt, and generate a citation-backed answer:

# Using local Ollama (default)
docpipe-maestro ask "How many days of leave does a first-year employee get?" --collection hr_policy

# Using a custom Ollama tag or cloud model / OpenAI-compatible base URL
docpipe-maestro ask "Summarize travel meal limits." --model ollama/qwen2.5:1.5b-instruct --collection hr_policy
export OPENAI_API_KEY="sk-..."
docpipe-maestro ask "Summarize travel meal allowance limits." --model openai/gpt-4o-mini --collection hr_policy
docpipe-maestro ask "Summarize travel policy." --model openai/my-local-model --api-base http://localhost:8000/v1 --collection hr_policy

# Safe thresholding: if no retrieved content matches threshold, short-circuits with a refusal
docpipe-maestro ask "What is the recipe for lasagna?" --threshold 0.7 --collection hr_policy

# Inspect retrieved context on stderr, request JSON telemetry on stdout, stream tokens live, or fail if truncated at max_tokens
docpipe-maestro ask "Summarize travel limits." --show-context --collection hr_policy
docpipe-maestro ask "What is the SEV-1 escalation window?" --searcher fts5_hybrid --rerank --collection hr_policy
docpipe-maestro ask "How many days of annual leave do employees get?" --stream --collection hr_policy
docpipe-maestro ask "Summarize travel limits." --json --fail-on-truncation --collection hr_policy

# Enforce local-only execution pre-network or scrub outbound remote prompts via --pii-policy
docpipe-maestro ask "Summarize travel limits." --local-only --collection hr_policy
docpipe-maestro ask "Summarize travel limits." --where source=travel_policy.md --collection hr_policy
docpipe-maestro ask "Summarize employee contacts." --model openai/gpt-4o-mini --pii-policy ./pii-policy.json --collection hr_policy

(By default, if a provider hits --max-tokens and sets finish_reason == "length", ask prints a Warning: on stderr; pass --fail-on-truncation to fail non-zero instead.)

8. Offline Retrieval Evaluation (docpipe-maestro eval)

Benchmark Phase 2 retrieval (Precision@K, Recall@K, MRR) against a labeled JSON fixture without mutating persisted manifests or indexes (see Retrieval Evaluation Baselines):

# Run curated 24-query policy benchmark with hybrid lexical + dense search
docpipe-maestro eval --fixture tests/fixtures/eval_policy_corpus.json --searcher fts5_hybrid

# Benchmark with local cross-encoder reranking enabled
docpipe-maestro eval --fixture tests/fixtures/eval_policy_corpus.json --searcher fts5_hybrid --rerank

# Customize cutoff values, filter subset by metadata, inspect raw child chunks, or emit JSON
docpipe-maestro eval --fixture tests/fixtures/eval_policy_corpus.json --where profile=POLICY --k 1 3 5 --no-expand-parent --json

(Target collection resolves in order: explicit --collection > fixture collection > docpipe.toml collection > "default".)


Python Library API Guide

See the complete Python API Reference for every signature, parameter default, config constant, and exception class.

Phase 1: Ingestion & Lifecycle Workflow

from pathlib import Path
from docpipe_maestro import PiiBlockedError, PiiPolicy, ingest, reindex, remove, status, sync

pii_policy: PiiPolicy = {
    "types": {
        "EMAIL_ADDRESS": "tokenize",
        "IBAN_CODE": "redact",
        "US_SSN": "block",
    },
    "token_salt_env": "DOCPIPE_PII_SALT",  # omit to use process-random fallback salt + warning
    "egress": "redact",
}

# 1. Ingest a document or directory with pre-chunking PII safeguards
try:
    result = ingest(
        Path("./policies"),
        profile="POLICY",  # "DEFAULT", "POLICY", or custom ChunkingProfile
        collection="handbook",
        persist_dir=".docpipe",
        embed_model="ollama/nomic-embed-text",
        pii_policy=pii_policy,
        local_only=True,
    )
    print(
        f"Ingested {len(result.ingested)} files "
        f"({result.chunks_written} children, {result.parents_written} parents)."
    )
except PiiBlockedError as exc:
    # Exception reports source & blocked entity type(s) without printing raw PII values
    print(f"Aborted cleanly on {exc.source}: {exc.blocked_entities}")

# 2. Synchronize folder after changes (adds, edits, deletions)
sync_result = sync(
    Path("./policies"),
    profile="POLICY",
    collection="handbook",
)
print(
    f"Added: {len(sync_result.added)}, Updated: {len(sync_result.updated)}, "
    f"Skipped: {len(sync_result.skipped)}, Removed: {len(sync_result.removed)}"
)

# 3. Check collection metadata and tracked documents
coll_status = status(collection="handbook")
print(
    f"Locked embed model: {coll_status.embed_model}, "
    f"Tracked files: {coll_status.total_sources}, "
    f"Children: {coll_status.total_children}"
)

# 4. Remove a document from the index
removed = remove("outdated_policy.md", collection="handbook")

# 5. Reindex collection with a new embedding model
reindexed_count = reindex(
    embed_model="openai/text-embedding-3-small",
    collection="handbook",
)

Phase 2: Semantic Retrieval & Grounded RAG Workflow

from docpipe_maestro import answer, build_chat_messages, retrieve

question = "What is the travel meal reimbursement rate?"

# 1. Retrieve deduplicated parent sections (pass searcher="fts5_hybrid" for dense + BM25 RRF)
retrieved_sections = retrieve(
    question=question,
    collection="handbook",
    top_k=5,
    threshold=0.35,  # Normalized similarity / RRF score [0.0, 1.0]
    expand_parent=True,  # Expand child matches to full parent section
    searcher="default",  # "default" or "fts5_hybrid"
    rerank=False,  # Set True (with `[rerank]` extra installed) for cross-encoder rescoring
    where={"profile": "POLICY"},  # Optional chunk/manifest metadata filter
)

for section in retrieved_sections:
    print(f"[{section.score:.2f}] {section.metadata['source']} - {section.metadata['section']}")
    print(section.text)

# 2. Build role-separated ChatML messages (Pure function, zero network calls)
#    (build_prompt() also remains available for single-string prompts.)
messages = build_chat_messages(question=question, sections=retrieved_sections)

# 3. Generate a grounded answer (pass stream=True for synchronous Iterator[str] chunks,
#    or use answer_detailed() for finish_reason/token telemetry)
response = answer(
    prompt=messages,
    model="ollama/llama3.2",
    temperature=0.0,  # Omit (None) for reasoning models that reject temperature
    max_tokens=512,  # Omit (None) or pass model_kwargs={"reasoning_effort": "low"} for reasoning models
    timeout=60.0,
    has_context=bool(retrieved_sections),  # Short-circuits with refusal if context is empty
)
print("Answer:\n", response)

# Incremental streaming with identical preflight/egress safety before the first pull:
for delta in answer(
    messages, model="ollama/llama3.2", has_context=bool(retrieved_sections), stream=True
):
    print(delta, end="", flush=True)
print()

Retrieval Evaluation (docpipe_maestro.eval)

from docpipe_maestro.eval import evaluate, normalize_k_values, precision_at_k, recall_at_k

report = evaluate(
    "tests/fixtures/eval_policy_corpus.json",
    collection="eval_policy",
    persist_dir=".docpipe",
    k_values=normalize_k_values([10, 3, 3, 1]),  # -> (1, 3, 10)
    searcher="fts5_hybrid",
    where={"profile": "POLICY"},  # Optional chunk/manifest metadata filter
)
print(report["aggregate"]["mrr"], report["aggregate"]["recall_at_k"])

Opt-In Structured Logging (docpipe_maestro.log)

Library calls stay silent by default (NullHandler only). Attach a formatter on "docpipe_maestro" when integrating with your application logs:

import logging
from docpipe_maestro.log import JsonFormatter, correlation_id, get_logger

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger = get_logger()
logger.addHandler(handler)
logger.setLevel(logging.INFO)

with correlation_id("req-42"):
    # All internal stage.end telemetry events share correlation_id="req-42"
    ...

Core Data Shapes & Exceptions

Exported from docpipe_maestro (docpipe_maestro.types):

  • Chunk: Base unit (id, text, parent_id, metadata: ChunkMetadata).
  • EmbeddedChunk: Chunk with dense embedding (vector: list[float]).
  • RetrievedChunk: EmbeddedChunk with normalized similarity or RRF score (score: float in [0.0, 1.0]).
  • GenerationResult: Detailed LLM completion result (text, model, finish_reason, truncated, prompt_tokens, completion_tokens, reasoning_tokens, total_tokens, elapsed_seconds).
  • IngestResult: Summary (collection, embed_model, ingested, skipped, chunks_written, parents_written).
  • SyncResult: Summary (collection, embed_model, added, updated, skipped, removed, dry_run, estimated_children, estimated_parents, estimated_tokens, estimated_cost_usd).
  • CollectionStatus: Manifest overview (collection, embed_model, embed_dim, total_sources, total_children, total_parents, sources).
  • SourceRecord: Per-file manifest tracking record (source, content_hash, doc_version, size, mtime, ingested_at, embed_model, profile, chunk_ids, parent_ids).
  • Exceptions: DocpipeError (base) with UnsupportedFormatError, MissingExtraError, ModelMismatchError, MissingCredentialError, PiiBlockedError, and RegistryError.
  • Privacy & Extensibility: PiiPolicy (TypedDict) passed programmatically to ingest(), sync(), answer(), and answer_detailed(), plus lazy root @register_pii_recognizer (docpipe_maestro.types.PiiRecognizer) for custom entity patterns.

Security, Privacy & Grounding Notes

  • Trusted local corpus assumption: Like most local ingestion tools, docpipe-maestro ingest processes files inside the target directory you specify. Only ingest trusted documents and avoid pointing ingest/sync at directories containing symlinks to sensitive files or untrusted HTML/PDF inputs.
  • Credentials & transport security: API keys are never accepted in docpipe.toml; remote custom endpoints require https:// (http:// is permitted only for loopback hosts), and error handlers scrub regex key patterns as well as loaded environment secrets (from None) so credentials never leak in exceptions.
  • --local-only & outbound PII egress guard: Pass --local-only (local_only=True) to reject non-local models and remote --api-base URLs prior to network requests. When calling remote LLMs with pii_policy, guard_egress() scrubs or blocks outbound prompt strings and role-separated ChatML messages before dispatch, and Phase 1 writes zero-raw-PII counts/metadata to <persist_dir>/manifests/<collection>.audit.jsonl under 0o700/0o600 permissions.
  • Opt-in structured telemetry (docpipe_maestro.log / --log-json): Library calls stay silent by default (NullHandler only on "docpipe_maestro", root logger and stdout untouched). CLI flags --log-json and --log-level {DEBUG,INFO,WARNING,ERROR} (CLI-only; ignored in docpipe.toml) stream correlated stage.end events exclusively to stderr with traceback suppression and zero raw documents, questions, prompts, answers, PII, or credentials at INFO or DEBUG.
  • Honest scope & accuracy limitations: Built-in pattern/checksum recognizers (EMAIL_ADDRESS, IBAN_CODE, CREDIT_CARD, US_SSN, IP_ADDRESS, PHONE_NUMBER, ES_DNI) and optional [pii] Presidio NER heuristics can experience both false positives and false negatives depending on language, formatting, custom identifiers, and OCR/PDF text noise. These safeguards reduce risk but are technical controls—not a guarantee of exhaustive redaction or regulatory certification. See SECURITY.md for full threat-model notes.
  • Reduce hallucinations with --threshold: Set a similarity --threshold (e.g. 0.35–0.6 depending on embedding model) on retrieve() / docpipe-maestro ask so off-topic questions return zero matches and short-circuit deterministically ("Not found in the provided context.").
  • Protect your .docpipe/ store on shared hosts: Embedded ChromaDB files store plaintext document sections unless scrubbed beforehand; keep persist_dir on owner-restricted paths (chmod 700) and excluded from version control.

Safety & Verification

docpipe-maestro includes two mandatory automated release gates:

  1. Reverse Safety Test Gate (tests/test_reverse_safety.py, @pytest.mark.reverse): Enforces that every clause containing an exception marker (except, unless, not entitled, subject to, provided that) always retrieves a parent context containing the exception alongside the general rule.
  2. PII Safety Test Gate (tests/test_pii_safety.py, @pytest.mark.pii): Verifies offline on synthetic fixtures that blocked PII halts ingestion before database commit, redacted/tokenized spans never persist to chunks or outbound remote payloads, --local-only blocks remote endpoints pre-network, and .audit.jsonl logs zero raw PII.

Run the test suite and quality gates:

# Run linters, security rules, and formatting checks
uv run ruff check .
uv run ruff check --select S src/
uv run ruff format --check .

# Run static type checking (strict mode)
uv run mypy src tests

# Run all unit, integration, and safety tests with branch coverage offline
LITELLM_LOCAL_MODEL_COST_MAP=true uv run pytest -v --cov

# Verify reverse and PII safety gates individually
uv run pytest -m reverse -v
uv run pytest -m pii -v
uv run pytest -m integration -v

Repository Structure

docpipe-maestro/
├── .github/                 # CI (ci.yml), release publishing (release.yml) & issue/PR templates
├── .specify/
│   └── memory/
│       └── constitution.md  # Non-negotiable architectural & safety principles
├── src/docpipe_maestro/
│   ├── __init__.py          # Public API exports & lazy pipeline/answer imports
│   ├── __main__.py          # Enables `python -m docpipe_maestro` CLI execution
│   ├── py.typed             # PEP 561 typed package marker
│   ├── cli.py               # `docpipe-maestro` CLI + TOML config loader
│   ├── config.py            # Default constants & immutable chunking profiles
│   ├── _crossrefs.py        # Shared phase-neutral cross-reference matching & excerpt helpers
│   ├── _filters.py          # Shared phase-neutral metadata filter validation, manifest resolution & SQL compiler
│   ├── _pii.py              # Shared phase-neutral PII recognizers, HMAC tokenizer & lazy NER
│   ├── _registry.py         # Lazy Strategy + Registry core (`register_*` decorators)
│   ├── _resolve.py          # Format-to-parser key resolution helper
│   ├── _stages.py           # Top-level stage re-exports and protocols
│   ├── log.py               # Opt-in structured JSON/text telemetry, NullHandler logger & ContextVar correlation IDs
│   ├── report.py            # Offline single-file HTML Document Integrity & Chunk Audit Report exporter
│   ├── embedder/            # Batched LiteLLM embedding adapter, locality checks & registry
│   ├── manifest.py          # Atomic sha256/mtime manifest, model lock & .audit.jsonl logger
│   ├── store/               # ChromaDB vector store (`chroma.py`) & SQLite FTS5 (`lexical.py`)
│   ├── types.py             # Dataclasses, TypedDicts & DocpipeError hierarchy
│   ├── ingest/              # Phase 1: intake/, clean/, enricher/ (`pii`, `crossrefs`), chunker/, pipeline
│   ├── query/               # Phase 2: searcher/, reranker/, egress.py, retriever, prompts, answer
│   └── eval/                # Phase 2: offline metrics (`metrics.py`) & read-only runner (`harness.py`)
├── tests/                   # Unit, integration, security, reverse & PII-safety suites
├── documents/               # Synthetic sample policy corpus for local demos
├── docs/                    # Architecture, API, CLI, baselines, dev guide & roadmap
├── specs/                   # Feature specification, plan & contracts
├── .env.example             # Template for local/cloud provider environment variables
├── .gitattributes           # Cross-OS LF line-ending enforcement (keeps binary PDFs/DOCX untouched)
├── .pre-commit-config.yaml  # Commitizen Conventional Commits & Ruff hook configuration
├── CHANGELOG.md             # Release history
├── CONTRIBUTING.md          # Contribution workflow & quality gate checklist
├── LICENSE                  # MIT License
├── PRODUCT.md               # Product specification & requirements baseline
├── README.md                # Project overview, quickstart & guides
├── SECURITY.md              # Vulnerability reporting & privacy model
└── pyproject.toml           # PEP 621 / 735 packaging, ruff, mypy & pytest config

Contributing

Contributions are welcome! Please read CONTRIBUTING.md and the Development & Testing Guide before opening a pull request. In brief:

  1. Follow Conventional Commits (feat:, fix:, docs:, test:, chore:) verified with uv run cz check.
  2. Pass all CI quality gates (ruff format --check ., ruff check ., ruff check --select S src/, mypy src tests, pytest -m reverse -v, pytest -m pii -v, and pytest -v --cov with branch coverage >= 80%).
  3. Keep all Phase 1 and Phase 2 library modules free of config/policy-file reading and external UI/server concerns.

For security vulnerabilities, see SECURITY.md.


Documentation


License

This project is licensed under the MIT License.

Metadata

Release files for docpipe-maestro 0.4.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for docpipe-maestro 0.4.1
File Size Uploaded Provenance Actions
docpipe_maestro-0.4.1.tar.gz 380.4 kB With provenance Details for docpipe_maestro-0.4.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for docpipe-maestro 0.4.1
File Interpreter ABI Platform Actions
docpipe_maestro-0.4.1-py3-none-any.whl Python 3 none any Details for docpipe_maestro-0.4.1-py3-none-any.whl

Total release size: 543.3 kB

Release files / docpipe_maestro-0.4.1.tar.gz

Download URL docpipe_maestro-0.4.1.tar.gz
Size 380.4 kB
Tags Source
SHA-256 checksum
How to use checksums
ab60aabe81f9ea663bb69439e859369f52152d090c27b7a41522984937e7f0a9
BLAKE2b-256 checksum
How to use checksums
ec802e1e8140494684d586238e739b9d1abcf23f7f93ec8354fac46cff34b151
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Release files / docpipe_maestro-0.4.1-py3-none-any.whl

Download URL docpipe_maestro-0.4.1-py3-none-any.whl
Size 162.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b2b6c50913f8d80bbe6c05bd69da512dea678155129b84bc315aa2e03b570eaa
BLAKE2b-256 checksum
How to use checksums
c9abe44bab9c4508446f3f36f2494f6548c4a50ff0b4cf23480d54bdb1938040
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Security for docpipe-maestro 0.4.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for docpipe_maestro-0.4.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page