docpipe-maestro
"docpipe-maestro is a minimal RAG pipeline: it ingests any document, chunks it correctly, embeds it, stores it, retrieves the right sections, and optionally generates a grounded answer. It is the engine — not the app, not the framework."
docpipe-maestro is a provider-agnostic, structure-aware Python RAG (Retrieval-Augmented Generation) pipeline library and docpipe-maestro CLI. It solves the data-preparation and retrieval failure modes common in standard fixed-size chunkers: preserving document hierarchy, refusing to split legal or policy clauses mid-sentence, resolving intra-document cross-references, and guaranteeing that policy exceptions always stay attached to their rules.
Table of Contents
- Why docpipe-maestro?
- 60-Second Quickstart
- Key Architecture & Core Concepts
- Supported Formats & Limitations
- Installation Guide
- Configuration Guide
- Quickstart: CLI Usage Guide
- 1. Ingest Documents (
docpipe-maestro ingest) - 2. Keep Collections in Sync (
docpipe-maestro sync) - 3. Inspect Collection Status (
docpipe-maestro status) - 4. Remove a Source Document (
docpipe-maestro remove) - 5. Migrate Embedding Models (
docpipe-maestro reindex) - 6. Semantic Search Without an LLM (
docpipe-maestro query) - 7. Grounded Question Answering (
docpipe-maestro ask) - 8. Offline Retrieval Evaluation (
docpipe-maestro eval)
- 1. Ingest Documents (
- Python Library API Guide
- Security, Privacy & Grounding Notes
- Safety & Verification
- Repository Structure
- Contributing
- Documentation
- License
Why docpipe-maestro?
| Common RAG Failure Mode | How docpipe-maestro Solves It |
|---|---|
| Orphaned exceptions ("20 days leave" retrieved without "except probationary hires") | Parent-child retrieval + POLICY clause preservation searches fine-grained child chunks and returns deduplicated full parent sections, enforced by the mandatory Reverse Safety Gate (tests/test_reverse_safety.py). |
| Broken intra-document references ("subject to Section 4.2" without knowing what Section 4.2 says) | POLICY CrossrefsEnricher (enrichers=("crossrefs",)) appends cycle-safe [Crossref ...] excerpts directly onto referring sections during Phase 1 ingestion (toggleable at query time via --no-crossrefs / follow_crossrefs=False). |
Exact identifiers & ranking misses in dense-only search (SEC-101, IR-15M) |
Hybrid search + child-first reranking pairs collocated SQLite FTS5 BM25 + dense vector RRF (fts5_hybrid) with optional local cross-encoder reranking (--rerank / rerank=True). |
| Cross-corpus noise across files, sections, profiles, or dates | Unified where metadata & manifest filtering compiles validated filter operators ($eq, $ne, $in, $nin, $gt, $gte, $lt, $lte, $and, $or) over chunk fields and manifest attributes (profile, ingested_after, ingested_before) with ChromaDB and FTS5 parity and zero re-ingestion. |
| Orphaned vectors when files shrink or change | Transactional doc_version staging writes new chunks, flips the JSON manifest atomically, and deletes stale versions afterward (with startup sweep_uncommitted() recovery). Failed runs leave the prior version intact. |
| Silent embedding vector-space corruption | Manifest model lock pins each collection to one embed_model and raises ModelMismatchError unless migrated explicitly via reindex. |
| Hallucinations when nothing relevant matches | Deterministic empty-context refusal returns "Not found in the provided context." without calling the LLM (ask checks retrieved hits automatically; answer()/answer_detailed() short-circuit when has_context=False). |
| Sensitive PII stored or sent to remote LLMs | Opt-in pre-chunking PiiEnricher + outbound guard_egress() redact, HMAC-tokenize, or block entities cleanly (PiiBlockedError), log zero-raw-PII .audit.jsonl records (0o700/0o600), and enforce --local-only pre-network across embeddings and generation. |
| Hidden HTML prompt injections & unsafe archives | Hardened intake strips hidden DOM elements (display:none, hidden, aria-hidden, <template>), enforces DOCX uncompressed size caps, and skips directory symlinks. |
| Blind retrieval regressions, vendor lock-in & library bloat | Offline eval harness, Strategy registries & Library Purity deliver reproducible MRR / Precision@k / Recall@k checks, 9 @register_* extension points, NullHandler-only docpipe_maestro.log (--log-json stderr telemetry), zero config-file reads in library code, and LiteLLM routing with zero hardcoded provider registry. |
60-Second Quickstart
Run an entirely local pipeline using Ollama (nomic-embed-text + llama3.2):
# 1. Install from PyPI with all format parsers
pip install "docpipe-maestro[all]"
# 2. Pull local Ollama models (one-time)
ollama pull nomic-embed-text
ollama pull llama3.2
# 3. Ingest a folder of policies and ask a grounded question
docpipe-maestro ingest ./policies --profile POLICY --collection handbook
docpipe-maestro ask "When are employees not entitled to overtime?" --collection handbook --threshold 0.35
(Running from a cloned source repository? Prefix CLI commands with uv run docpipe-maestro ... or uv run python -m docpipe_maestro ....)
Key Architecture & Core Concepts
flowchart LR
subgraph P1["Phase 1: Offline Ingestion (Atomic & Transactional)"]
A[Source Files] --> B[Intake & Format Detect]
B --> C[Clean & Boilerplate Strip]
C --> D["Enrichers (PII -> Crossrefs)"]
D --> E[Structure-Aware Chunker]
E --> F["local_only Check + Batched Embeddings"]
F --> G[(ChromaDB + FTS5 + Manifest + .audit.jsonl)]
end
subgraph P2["Phase 2: Online Query (Read-Only)"]
Q[Question] --> QE["local_only Check + Query Embed"]
QE --> S["Child Search (Dense or fts5_hybrid RRF)"]
S --> RR["Optional Rerank (Cross-Encoder Sigmoid)"]
RR --> PE["Parent Expansion & max(score) Dedupe"]
PE --> BP["Pure build_prompt() / build_chat_messages()"]
BP -->|results > 0| EG["Egress Guard (local_only + PII Policy)"]
EG --> LLM["Optional answer() via LiteLLM"]
BP -->|empty results| REF["Deterministic Refusal (No LLM Call)"]
end
- Two-Phase Separation:
- Phase 1 (Offline Ingestion): Files $\to$ Parse $\to$ Clean $\to$ Enrich (
pii$\to$crossrefs) $\to$ Structure-Aware Chunking $\to$ Embeddings $\to$ Vector Database +.audit.jsonl. Correctness and crash safety matter. - Phase 2 (Online Query): Question $\to$ Embed $\to$ Retrieve (Match Children, Return Deduplicated Parents) $\to$ Build Prompt $\to$ Egress Guard $\to$ Optional Grounded LLM Generation. Read-only and low-latency.
- Phase 1 (Offline Ingestion): Files $\to$ Parse $\to$ Clean $\to$ Enrich (
- Search Small, Return Big (Parent-Child Retrieval):
- Small child chunks (individual clauses/paragraphs prefixed with
[Source: ... | Section: ...]) are embedded and searched for high precision. - Large parent sections (complete contextual sections) are stored with zero-vector placeholders and returned to the prompt.
- Small child chunks (individual clauses/paragraphs prefixed with
- Exception Preservation & Reverse Safety Gate:
- In policy and legal documents, a rule without its exception is actively misleading (e.g., "Employees receive 20 days leave" without "unless probationary"). In
POLICYprofile, clauses are kept intact even when exceeding target chunk size, and parent sections ensure exceptions always accompany their rules.
- In policy and legal documents, a rule without its exception is actively misleading (e.g., "Employees receive 20 days leave" without "unless probationary"). In
- Crash-Safe Transactional Synchronization:
- Ingestion maintains an atomic database-collocated manifest (
.docpipe/manifests/<collection>.json) trackingsha256content hashes,mtime, anddoc_versionprefixes. - Modifying a document writes new versioned chunks, commits the manifest atomically, and deletes stale versions afterward, preventing orphaned clauses when documents shrink.
- Ingestion maintains an atomic database-collocated manifest (
- Embedding Model Lock:
- The manifest locks the collection to the embedding model used during ingestion. Attempting to ingest or query with a different model raises
ModelMismatchErrorrather than silently mixing incompatible vector spaces. Model migration is handled cleanly viareindex.
- The manifest locks the collection to the embedding model used during ingestion. Attempting to ingest or query with a different model raises
Supported Formats & Limitations
| Format | Extensions | Structural & Content Extraction | Extra Required |
|---|---|---|---|
| Markdown | .md, .markdown |
ATX headings (#–######), code fence awareness (``` / ~~~ ignores # comments) |
Core install |
| Plain Text | .txt |
Heuristic Section / Article / Chapter / Part and numbered headings |
Core install |
| HTML | .html, .htm |
<h1–h6> headings, paragraphs, lists, tables (|), strips <script>/<style>/<nav>/<footer>/<noscript> |
[html] (beautifulsoup4) |
| Word | .docx |
Word Heading 1–Heading 6 styles, paragraphs, tables (|) |
[docx] (python-docx) |
.pdf |
Per-page text, 1-based page numbers, multi-page active heading carryover, boilerplate header/footer stripping | [pdf] (pypdf) |
- Extensionless files:
detect_format()inspects the first 4 KiB for%PDF-, ZIPword/document.xml, HTML root tags (<!doctype html/<html), Markdown headings, or valid UTF-8 text. - Current scope & limitations:
- Image-only / scanned PDFs without an embedded text layer yield empty text (OCR is out of scope).
- Legacy binary
.docfiles are not supported (.docxonly). docpipe-maestrois a library and CLI engine — it deliberately excludes web servers, auth layers, streaming UIs, and multi-tenant isolation.
Installation Guide
docpipe-maestro requires Python 3.10+.
Core Install
The core installation includes Markdown (.md, .markdown) and plain text (.txt) parsing, embedding generation, ChromaDB vector storage, and semantic retrieval:
# Install from PyPI
pip install docpipe-maestro
# Or install the latest development head from GitHub:
# pip install "git+https://github.com/ivnmoreno/docpipe-maestro.git"
Optional Extras
Generation (answer(), ask) is available in the base install — no extra needed. Install optional extras for additional document format parsers:
| Extra | Purpose | Dependencies Installed |
|---|---|---|
pdf |
Extract text, page numbers, and heuristic section headings (Section X.Y) from PDF files |
pypdf>=6.21.3 (BSD-3-Clause) |
docx |
Extract text, style-based headings (Heading 1–Heading 6), and tables from Word documents |
python-docx>=1.1.0 |
html |
Strip boilerplate and parse <h1–h6> headings from web pages / HTML exports |
beautifulsoup4>=4.12.0 |
all |
Installs all optional document format parsers (pdf, docx, html) |
pypdf, python-docx, beautifulsoup4 |
rerank |
Second-stage local neural cross-encoder reranking (BAAI/bge-reranker-base default) |
sentence-transformers>=3.0.0 |
pii |
Optional multi-language NER PII recognition alongside deterministic recognizers | presidio-analyzer>=2.2.0, presidio-anonymizer>=2.2.0 |
Note on Structure Extraction: Markdown, plain text, HTML, DOCX, and PDF extract headings, clauses, and pagination metadata cleanly without external SaaS parsers. Note that
[all]installs document parsers only; include[rerank]and/or[pii]alongside it ([all,rerank,pii]) when enabling reranking or Presidio NER. For offline[pii]usage, pre-install the required local spaCy models (e.g.en_core_web_sm,es_core_news_sm).
Install specific extras:
# PDF support only
pip install "docpipe-maestro[pdf]"
# PDF and DOCX support
pip install "docpipe-maestro[pdf,docx]"
# Full installation with all document format parsers
pip install "docpipe-maestro[all]"
# Parsers + local cross-encoder reranking + optional NER PII detection
pip install "docpipe-maestro[all,rerank,pii]"
Development Setup
To contribute or run tests locally (uses PEP 735 [dependency-groups]):
# Clone the repository
git clone https://github.com/ivnmoreno/docpipe-maestro.git
cd docpipe-maestro
# Option A: Using uv (recommended)
uv sync --all-extras --group dev
# Option B: Using pip 24.1+ (PEP 735 dependency groups)
pip install --group dev -e ".[all]"
pre-commit install --hook-type commit-msg
Note on Versioning & Commitizen: Package versions are derived dynamically from Git tags via
hatch-vcs. Useuv run cz check(to validate Conventional Commits) anduv run cz changelog(to updateCHANGELOG.md), rather thancz bump.
Configuration Guide
docpipe-maestro enforces strict Library Purity: no library module reads external configuration files. Configuration is passed explicitly via function arguments or provider environment variables. Only the docpipe-maestro CLI optionally reads a --config TOML file.
1. Local Execution (Default: Zero Cloud Accounts Required)
Out of the box, docpipe-maestro defaults to local execution with Ollama:
- Default Embeddings Model:
ollama/nomic-embed-text - Default Generation Model:
ollama/llama3.2
Make sure Ollama is installed and running (http://localhost:11434), then pull the default models:
ollama pull nomic-embed-text
ollama pull llama3.2
Tip — Choosing a local generation model: For multi-section policy synthesis (especially on CPU or limited RAM),
ollama/qwen2.5:1.5b-instructorollama/qwen2.5:3bfollows retrieved context more reliably thanollama/llama3.2(which can over-refuse paraphrased queries) and<1Bmodels (which are too small for grounded synthesis). Override the default with--model ollama/qwen2.5:1.5b-instructor setmodel = "ollama/qwen2.5:1.5b-instruct"indocpipe.toml.
Air-gapped / strict zero-egress environments: By default,
litellmattempts to download its public model-cost JSON map from GitHub when first imported before falling back to its bundled copy. Setexport LITELLM_LOCAL_MODEL_COST_MAP=truein your shell to prevent any startup HTTP call on import.
2. Cloud Provider API Keys (.env or Shell Environment)
Copy the tracked template .env.example to .env (git-ignored) and restrict permissions:
cp .env.example .env && chmod 600 .env
Or export provider environment variables directly in your shell (shell variables always take precedence over .env):
# Groq serverless inference
export GROQ_API_KEY="your-groq-api-key"
# Together.ai serverless inference (TOGETHER_API_KEY alias is also recognized by LiteLLM)
export TOGETHERAI_API_KEY="your-together-api-key"
# OpenAI / Anthropic / Google Gemini
export OPENAI_API_KEY="your-openai-api-key"
export ANTHROPIC_API_KEY="your-anthropic-api-key"
export GEMINI_API_KEY="your-gemini-api-key"
# Air-gapped / offline startup cost-map safety
export LITELLM_LOCAL_MODEL_COST_MAP="true"
You can then pass any provider/model string directly to --embed-model or --model (e.g., groq/llama-3.3-70b-versatile, together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo, openai/gpt-4o-mini, gemini/gemini-2.5-flash). Note that the CLI loads .env strictly from the --config directory or current working directory (never walking parent directories), while direct Python API callers configure os.environ.
3. Local Persistence Storage Directory
By default, data is stored in ./.docpipe:
.docpipe/
├── chroma.sqlite3 # ChromaDB embedded vector database
├── <uuid>/ # ChromaDB vector segments and HNSW index files
├── lexical/
│ └── <collection>.db # Collocated SQLite FTS5 BM25 lexical index (`fts5_hybrid`)
└── manifests/
├── <collection>.json # Atomic JSON manifest tracking file hashes, versions, and locked model
└── <collection>.audit.jsonl # Append-only zero-raw-PII audit log (written when PII policy is active)
Customize the persistence directory with --persist-dir on the CLI or persist_dir= in Python. Never commit .docpipe/ to version control if your corpus is private.
4. CLI Configuration File (docpipe.toml)
When --config is omitted (and --no-config is not passed), the CLI automatically loads ./docpipe.toml from the current working directory if present (non-recursively; never walking parent directories). Create an optional TOML configuration file to set default CLI options (never store API keys or secrets here — explicit CLI flags override --config / ./docpipe.toml, which override built-in defaults):
# docpipe.toml
collection = "my_kb"
persist_dir = "./data/docpipe_index"
embed_model = "ollama/nomic-embed-text"
model = "ollama/llama3.2"
profile = "POLICY"
top_k = 5
threshold = 0.4
searcher = "default" # "default" (dense vector) or "fts5_hybrid" (dense + SQLite FTS5 RRF)
# Optional second-stage cross-encoder reranking (requires `[rerank]` extra):
# rerank = false
# reranker = "cross_encoder"
# rerank_top_n = 20
# Optional online/gateway defaults for `docpipe-maestro ask` (CLI flags override these):
# api_base = "https://api.groq.com/openai/v1"
# temperature = 0.0
# max_tokens = 512 # Keep omitted (or >= 4096) for reasoning models
# timeout = 60.0
# fail_on_truncation = false
# Privacy & PII defaults (JSON/TOML path string or `[pii_policy]` table applied in CLI only):
# local_only = false
# pii_policy = "./pii-policy.json"
Pass an explicit config path with --config, or ignore ./docpipe.toml with --no-config:
docpipe-maestro query "What is the probation period?" --config docpipe.toml
docpipe-maestro ask "What is the probation period?" --no-config
Optional PII Policy File (--pii-policy)
When enabling --pii-policy ./pii-policy.json (or .toml) on ingest, sync, or ask, configure actions ("redact", "tokenize", "block") per entity type (EMAIL_ADDRESS, IBAN_CODE, CREDIT_CARD, US_SSN, IP_ADDRESS, PHONE_NUMBER, ES_DNI, plus optional [pii] Presidio ner: true):
{
"types": {
"EMAIL_ADDRESS": "tokenize",
"IBAN_CODE": "redact",
"US_SSN": "block"
},
"token_salt_env": "DOCPIPE_PII_SALT",
"egress": "redact"
}
(In docpipe.toml, set pii_policy = "./pii-policy.json" or define an inline [pii_policy] table. Use token_salt_env outside the repository for deterministic cross-run tokens.)
5. Chunking Profiles (DEFAULT, POLICY, and Custom Profiles)
Built-in profiles are immutable presets (docpipe_maestro.config) calibrated for specific document types:
| Parameter | DEFAULT Profile |
POLICY Profile |
|---|---|---|
| Intended For | Notes, general prose, articles, documentation | Handbooks, compliance policies, HR manuals, contracts |
| Target Chunk Size | ~512 tokens | ~1000 tokens |
| Chunk Overlap | ~64 tokens (scoped within section) | ~80 tokens (scoped within section) |
| Clause Preservation | Disabled (splits on sentences/paragraphs) | Enabled (never splits inside detected numbered clauses; PDF clause detection is heuristic) |
| Cross-Reference Resolution | Disabled | Enabled (resolves §2.1, Section 4, Clause 3.2) |
Enrichers (enrichers) |
() (skipped) |
("crossrefs",) |
| Parent-Child Expansion | Enabled | Enabled |
Note on PDF Clause Detection: Markdown, DOCX, and HTML headings/numbering provide explicit structural boundaries, whereas PDF clause detection uses extracted-text heuristics (
1.1,Clause 2,(a)layout cues) and depends on clean PDF text extraction order.
In Python, you can also pass a custom ChunkingProfile instance directly to ingest(..., profile=custom_profile) or sync(...), and configure ordered post-clean enrichers via profile.enrichers and @register_enricher.
Quickstart: CLI Usage Guide
Run docpipe-maestro --help to see all available subcommands (global flags such as --config, --no-config, --collection, --persist-dir, --local-only, --log-json, and --log-level work either before or after any subcommand):
usage: docpipe-maestro [-h] [--version] [--config CONFIG] [--no-config]
[--collection COLLECTION] [--persist-dir PERSIST_DIR]
[--local-only | --no-local-only]
[--pii-policy PII_POLICY] [--log-json]
[--log-level {DEBUG,INFO,WARNING,ERROR}]
{ingest,sync,status,remove,reindex,query,ask,eval} ...
See the full CLI Reference for all flags, TOML keys, exit codes, and output schemas.
1. Ingest Documents (docpipe-maestro ingest)
Ingest a single file or an entire directory tree atomically (if any file fails validation, no records are committed):
# Ingest with default profile
docpipe-maestro ingest ./docs --collection knowledge_base
# Ingest policy documents with clause-preservation and cross-reference resolution
docpipe-maestro ingest ./company_policies --profile POLICY --collection hr_policy
# Ingest strictly locally with pre-chunking PII scrubbing/blocking and .audit.jsonl output
docpipe-maestro ingest ./company_policies --local-only --pii-policy ./pii-policy.json --collection hr_policy
# Force re-ingestion even if files have not changed on disk
docpipe-maestro ingest ./docs --force --collection knowledge_base
2. Keep Collections in Sync (docpipe-maestro sync)
Reconcile a target directory against the manifest (adds new files, skips unchanged files via mtime/sha256, updates modified files, and purges deleted files):
docpipe-maestro sync ./company_policies --profile POLICY --collection hr_policy
Example output:
Synced 'hr_policy': added=1, updated=2, skipped=10, removed=1.
3. Inspect Collection Status (docpipe-maestro status)
View indexed files, active doc_version hashes, chunk counts, and locked embedding model:
docpipe-maestro status --collection hr_policy
Example output:
Collection: hr_policy | Model: ollama/nomic-embed-text (dim=768) | Sources: 2 | Children: 18 | Parents: 6
- hr_leave_policy.md [4f1a29b01c3d] (10 children, 3 parents)
- travel_expenses_policy.md [a8c2194ef001] (8 children, 3 parents)
Use --json for machine-readable output:
docpipe-maestro status --collection hr_policy --json
4. Remove a Source Document (docpipe-maestro remove)
Purge all child chunks, parent sections, and manifest entries for a specific file:
docpipe-maestro remove hr_leave_policy.md --collection hr_policy
5. Migrate Embedding Models (docpipe-maestro reindex)
Re-embed all existing child chunks stored in the database under a new embedding model, update parent vector dimensions, and re-lock the collection:
# Migrate collection to OpenAI embeddings
export OPENAI_API_KEY="sk-..."
docpipe-maestro reindex --model openai/text-embedding-3-small --collection hr_policy
6. Semantic Search Without an LLM (docpipe-maestro query)
Perform Phase 2 semantic retrieval to inspect scored passages and provenance without invoking a language model:
# Basic semantic query
docpipe-maestro query "How many vacation days do full-time employees accrue?" --collection hr_policy
# Restrict to top 3 sections with a minimum similarity threshold (0.0 to 1.0)
docpipe-maestro query "What is the flight reimbursement policy?" --top-k 3 --threshold 0.5 --collection hr_policy
# Combine dense vector similarity with SQLite FTS5 BM25 via Reciprocal Rank Fusion
# (ideal for exact error codes, identifiers, or clauses like "SEV-1" or "AES-256")
docpipe-maestro query "SEV-1 escalation window" --searcher fts5_hybrid --collection hr_policy
# Apply optional second-stage cross-encoder reranking over top candidate child chunks
# (requires `docpipe-maestro[rerank]`)
docpipe-maestro query "SEV-1 escalation window" --searcher fts5_hybrid --rerank --rerank-top-n 20 --collection hr_policy
# Return raw child chunks instead of expanded parent sections
docpipe-maestro query "annual leave" --no-expand-parent --collection hr_policy
# Scope search by document metadata or manifest profile/ingestion window (without re-ingestion)
docpipe-maestro query "annual leave" --where profile=POLICY --where page=2 --collection hr_policy
docpipe-maestro query "annual leave" --where ingested_after=2026-01-01 --collection hr_policy
# Emit structured JSON (for evaluation fixtures and scripts) or add --log-json telemetry on stderr
docpipe-maestro query "annual leave" --json --collection hr_policy
docpipe-maestro --log-json --log-level INFO query "annual leave" --collection hr_policy 2>trace.jsonl
Example output:
[1] score=0.8421 | Source: hr_leave_policy.md | Section: 3. Annual Leave
3.1 Full-time employees accrue 20 days of paid annual leave per calendar year.
3.2 Employees with less than one year of tenure accrue leave on a pro-rata basis.
------------------------------------------------------------
7. Grounded Question Answering (docpipe-maestro ask)
Retrieve relevant context, build a grounded prompt, and generate a citation-backed answer:
# Using local Ollama (default)
docpipe-maestro ask "How many days of leave does a first-year employee get?" --collection hr_policy
# Using a custom Ollama tag or cloud model / OpenAI-compatible base URL
docpipe-maestro ask "Summarize travel meal limits." --model ollama/qwen2.5:1.5b-instruct --collection hr_policy
export OPENAI_API_KEY="sk-..."
docpipe-maestro ask "Summarize travel meal allowance limits." --model openai/gpt-4o-mini --collection hr_policy
docpipe-maestro ask "Summarize travel policy." --model openai/my-local-model --api-base http://localhost:8000/v1 --collection hr_policy
# Safe thresholding: if no retrieved content matches threshold, short-circuits with a refusal
docpipe-maestro ask "What is the recipe for lasagna?" --threshold 0.7 --collection hr_policy
# Inspect retrieved context on stderr, request JSON telemetry on stdout, or fail if truncated at max_tokens
docpipe-maestro ask "Summarize travel limits." --show-context --collection hr_policy
docpipe-maestro ask "What is the SEV-1 escalation window?" --searcher fts5_hybrid --rerank --collection hr_policy
docpipe-maestro ask "Summarize travel limits." --json --fail-on-truncation --collection hr_policy
# Enforce local-only execution pre-network or scrub outbound remote prompts via --pii-policy
docpipe-maestro ask "Summarize travel limits." --local-only --collection hr_policy
docpipe-maestro ask "Summarize travel limits." --where source=travel_policy.md --collection hr_policy
docpipe-maestro ask "Summarize employee contacts." --model openai/gpt-4o-mini --pii-policy ./pii-policy.json --collection hr_policy
(By default, if a provider hits --max-tokens and sets finish_reason == "length", ask prints a Warning: on stderr; pass --fail-on-truncation to fail non-zero instead.)
8. Offline Retrieval Evaluation (docpipe-maestro eval)
Benchmark Phase 2 retrieval (Precision@K, Recall@K, MRR) against a labeled JSON fixture without mutating persisted manifests or indexes (see Retrieval Evaluation Baselines):
# Run curated 24-query policy benchmark with hybrid lexical + dense search
docpipe-maestro eval --fixture tests/fixtures/eval_policy_corpus.json --searcher fts5_hybrid
# Benchmark with local cross-encoder reranking enabled
docpipe-maestro eval --fixture tests/fixtures/eval_policy_corpus.json --searcher fts5_hybrid --rerank
# Customize cutoff values, filter subset by metadata, inspect raw child chunks, or emit JSON
docpipe-maestro eval --fixture tests/fixtures/eval_policy_corpus.json --where profile=POLICY --k 1 3 5 --no-expand-parent --json
(Target collection resolves in order: explicit --collection > fixture collection > docpipe.toml collection > "default".)
Python Library API Guide
See the complete Python API Reference for every signature, parameter default, config constant, and exception class.
Phase 1: Ingestion & Lifecycle Workflow
from pathlib import Path
from docpipe_maestro import PiiBlockedError, PiiPolicy, ingest, reindex, remove, status, sync
pii_policy: PiiPolicy = {
"types": {
"EMAIL_ADDRESS": "tokenize",
"IBAN_CODE": "redact",
"US_SSN": "block",
},
"token_salt_env": "DOCPIPE_PII_SALT", # omit to use process-random fallback salt + warning
"egress": "redact",
}
# 1. Ingest a document or directory with pre-chunking PII safeguards
try:
result = ingest(
Path("./policies"),
profile="POLICY", # "DEFAULT", "POLICY", or custom ChunkingProfile
collection="handbook",
persist_dir=".docpipe",
embed_model="ollama/nomic-embed-text",
pii_policy=pii_policy,
local_only=True,
)
print(
f"Ingested {len(result.ingested)} files "
f"({result.chunks_written} children, {result.parents_written} parents)."
)
except PiiBlockedError as exc:
# Exception reports source & blocked entity type(s) without printing raw PII values
print(f"Aborted cleanly on {exc.source}: {exc.blocked_entities}")
# 2. Synchronize folder after changes (adds, edits, deletions)
sync_result = sync(
Path("./policies"),
profile="POLICY",
collection="handbook",
)
print(
f"Added: {len(sync_result.added)}, Updated: {len(sync_result.updated)}, "
f"Skipped: {len(sync_result.skipped)}, Removed: {len(sync_result.removed)}"
)
# 3. Check collection metadata and tracked documents
coll_status = status(collection="handbook")
print(
f"Locked embed model: {coll_status.embed_model}, "
f"Tracked files: {coll_status.total_sources}, "
f"Children: {coll_status.total_children}"
)
# 4. Remove a document from the index
removed = remove("outdated_policy.md", collection="handbook")
# 5. Reindex collection with a new embedding model
reindexed_count = reindex(
embed_model="openai/text-embedding-3-small",
collection="handbook",
)
Phase 2: Semantic Retrieval & Grounded RAG Workflow
from docpipe_maestro import answer, build_chat_messages, retrieve
question = "What is the travel meal reimbursement rate?"
# 1. Retrieve deduplicated parent sections (pass searcher="fts5_hybrid" for dense + BM25 RRF)
retrieved_sections = retrieve(
question=question,
collection="handbook",
top_k=5,
threshold=0.35, # Normalized similarity / RRF score [0.0, 1.0]
expand_parent=True, # Expand child matches to full parent section
searcher="default", # "default" or "fts5_hybrid"
rerank=False, # Set True (with `[rerank]` extra installed) for cross-encoder rescoring
where={"profile": "POLICY"}, # Optional chunk/manifest metadata filter
)
for section in retrieved_sections:
print(f"[{section.score:.2f}] {section.metadata['source']} - {section.metadata['section']}")
print(section.text)
# 2. Build role-separated ChatML messages (Pure function, zero network calls)
# (build_prompt() also remains available for single-string prompts.)
messages = build_chat_messages(question=question, sections=retrieved_sections)
# 3. Generate a grounded answer (use answer_detailed() for finish_reason/token telemetry)
response = answer(
prompt=messages,
model="ollama/llama3.2",
temperature=0.0, # Omit (None) for reasoning models that reject temperature
max_tokens=512, # Omit (None) or pass model_kwargs={"reasoning_effort": "low"} for reasoning models
timeout=60.0,
has_context=bool(retrieved_sections), # Short-circuits with refusal if context is empty
)
print("Answer:\n", response)
Retrieval Evaluation (docpipe_maestro.eval)
from docpipe_maestro.eval import evaluate, normalize_k_values, precision_at_k, recall_at_k
report = evaluate(
"tests/fixtures/eval_policy_corpus.json",
collection="eval_policy",
persist_dir=".docpipe",
k_values=normalize_k_values([10, 3, 3, 1]), # -> (1, 3, 10)
searcher="fts5_hybrid",
where={"profile": "POLICY"}, # Optional chunk/manifest metadata filter
)
print(report["aggregate"]["mrr"], report["aggregate"]["recall_at_k"])
Opt-In Structured Logging (docpipe_maestro.log)
Library calls stay silent by default (NullHandler only). Attach a formatter on "docpipe_maestro" when integrating with your application logs:
import logging
from docpipe_maestro.log import JsonFormatter, correlation_id, get_logger
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger = get_logger()
logger.addHandler(handler)
logger.setLevel(logging.INFO)
with correlation_id("req-42"):
# All internal stage.end telemetry events share correlation_id="req-42"
...
Core Data Shapes & Exceptions
Exported from docpipe_maestro (docpipe_maestro.types):
Chunk: Base unit (id,text,parent_id,metadata: ChunkMetadata).EmbeddedChunk:Chunkwith dense embedding (vector: list[float]).RetrievedChunk:EmbeddedChunkwith normalized similarity or RRF score (score: floatin[0.0, 1.0]).GenerationResult: Detailed LLM completion result (text,model,finish_reason,truncated,prompt_tokens,completion_tokens,reasoning_tokens,total_tokens,elapsed_seconds).IngestResult: Summary (collection,embed_model,ingested,skipped,chunks_written,parents_written).SyncResult: Summary (collection,embed_model,added,updated,skipped,removed).CollectionStatus: Manifest overview (collection,embed_model,embed_dim,total_sources,total_children,total_parents,sources).SourceRecord: Per-file manifest tracking record (source,content_hash,doc_version,size,mtime,ingested_at,embed_model,profile,chunk_ids,parent_ids).- Exceptions:
DocpipeError(base) withUnsupportedFormatError,MissingExtraError,ModelMismatchError,MissingCredentialError,PiiBlockedError, andRegistryError. - Privacy & Extensibility:
PiiPolicy(TypedDict) passed programmatically toingest(),sync(),answer(), andanswer_detailed(), plus lazy root@register_pii_recognizer(docpipe_maestro.types.PiiRecognizer) for custom entity patterns.
Security, Privacy & Grounding Notes
- Trusted local corpus assumption: Like most local ingestion tools,
docpipe-maestro ingestprocesses files inside the target directory you specify. Only ingest trusted documents and avoid pointingingest/syncat directories containing symlinks to sensitive files or untrusted HTML/PDF inputs. - Credentials & transport security: API keys are never accepted in
docpipe.toml; remote custom endpoints requirehttps://(http://is permitted only for loopback hosts), and error handlers scrub regex key patterns as well as loaded environment secrets (from None) so credentials never leak in exceptions. --local-only& outbound PII egress guard: Pass--local-only(local_only=True) to reject non-local models and remote--api-baseURLs prior to network requests. When calling remote LLMs withpii_policy,guard_egress()scrubs or blocks outbound prompt strings and role-separated ChatML messages before dispatch, and Phase 1 writes zero-raw-PII counts/metadata to<persist_dir>/manifests/<collection>.audit.jsonlunder0o700/0o600permissions.- Opt-in structured telemetry (
docpipe_maestro.log/--log-json): Library calls stay silent by default (NullHandleronly on"docpipe_maestro", root logger andstdoutuntouched). CLI flags--log-jsonand--log-level {DEBUG,INFO,WARNING,ERROR}(CLI-only; ignored indocpipe.toml) stream correlatedstage.endevents exclusively tostderrwith traceback suppression and zero raw documents, questions, prompts, answers, PII, or credentials atINFOorDEBUG. - Honest scope & accuracy limitations: Built-in pattern/checksum recognizers (
EMAIL_ADDRESS,IBAN_CODE,CREDIT_CARD,US_SSN,IP_ADDRESS,PHONE_NUMBER,ES_DNI) and optional[pii]Presidio NER heuristics can experience both false positives and false negatives depending on language, formatting, custom identifiers, and OCR/PDF text noise. These safeguards reduce risk but are technical controls—not a guarantee of exhaustive redaction or regulatory certification. See SECURITY.md for full threat-model notes. - Reduce hallucinations with
--threshold: Set a similarity--threshold(e.g.0.35–0.6depending on embedding model) onretrieve()/docpipe-maestro askso off-topic questions return zero matches and short-circuit deterministically ("Not found in the provided context."). - Protect your
.docpipe/store on shared hosts: Embedded ChromaDB files store plaintext document sections unless scrubbed beforehand; keeppersist_diron owner-restricted paths (chmod 700) and excluded from version control.
Safety & Verification
docpipe-maestro includes two mandatory automated release gates:
- Reverse Safety Test Gate (
tests/test_reverse_safety.py,@pytest.mark.reverse): Enforces that every clause containing an exception marker (except,unless,not entitled,subject to,provided that) always retrieves a parent context containing the exception alongside the general rule. - PII Safety Test Gate (
tests/test_pii_safety.py,@pytest.mark.pii): Verifies offline on synthetic fixtures that blocked PII halts ingestion before database commit, redacted/tokenized spans never persist to chunks or outbound remote payloads,--local-onlyblocks remote endpoints pre-network, and.audit.jsonllogs zero raw PII.
Run the test suite and quality gates:
# Run linters, security rules, and formatting checks
uv run ruff check .
uv run ruff check --select S src/
uv run ruff format --check .
# Run static type checking (strict mode)
uv run mypy src tests
# Run all unit, integration, and safety tests with branch coverage offline
LITELLM_LOCAL_MODEL_COST_MAP=true uv run pytest -v --cov
# Verify reverse and PII safety gates individually
uv run pytest -m reverse -v
uv run pytest -m pii -v
uv run pytest -m integration -v
Repository Structure
docpipe-maestro/
├── .github/ # CI (ci.yml), release publishing (release.yml) & issue/PR templates
├── .specify/
│ └── memory/
│ └── constitution.md # Non-negotiable architectural & safety principles
├── src/docpipe_maestro/
│ ├── __init__.py # Public API exports & lazy pipeline/answer imports
│ ├── __main__.py # Enables `python -m docpipe_maestro` CLI execution
│ ├── py.typed # PEP 561 typed package marker
│ ├── cli.py # `docpipe-maestro` CLI + TOML config loader
│ ├── config.py # Default constants & immutable chunking profiles
│ ├── _crossrefs.py # Shared phase-neutral cross-reference matching & excerpt helpers
│ ├── _filters.py # Shared phase-neutral metadata filter validation, manifest resolution & SQL compiler
│ ├── _pii.py # Shared phase-neutral PII recognizers, HMAC tokenizer & lazy NER
│ ├── _registry.py # Lazy Strategy + Registry core (`register_*` decorators)
│ ├── _resolve.py # Format-to-parser key resolution helper
│ ├── _stages.py # Top-level stage re-exports and protocols
│ ├── log.py # Opt-in structured JSON/text telemetry, NullHandler logger & ContextVar correlation IDs
│ ├── embedder/ # Batched LiteLLM embedding adapter, locality checks & registry
│ ├── manifest.py # Atomic sha256/mtime manifest, model lock & .audit.jsonl logger
│ ├── store/ # ChromaDB vector store (`chroma.py`) & SQLite FTS5 (`lexical.py`)
│ ├── types.py # Dataclasses, TypedDicts & DocpipeError hierarchy
│ ├── ingest/ # Phase 1: intake/, clean/, enricher/ (`pii`, `crossrefs`), chunker/, pipeline
│ ├── query/ # Phase 2: searcher/, reranker/, egress.py, retriever, prompts, answer
│ └── eval/ # Phase 2: offline metrics (`metrics.py`) & read-only runner (`harness.py`)
├── tests/ # Unit, integration, security, reverse & PII-safety suites
├── documents/ # Synthetic sample policy corpus for local demos
├── docs/ # Architecture, API, CLI, baselines, dev guide & roadmap
├── specs/ # Feature specification, plan & contracts
├── .env.example # Template for local/cloud provider environment variables
├── .gitattributes # Cross-OS LF line-ending enforcement (keeps binary PDFs/DOCX untouched)
├── .pre-commit-config.yaml # Commitizen Conventional Commits & Ruff hook configuration
├── CHANGELOG.md # Release history
├── CONTRIBUTING.md # Contribution workflow & quality gate checklist
├── LICENSE # MIT License
├── PRODUCT.md # Product specification & requirements baseline
├── README.md # Project overview, quickstart & guides
├── SECURITY.md # Vulnerability reporting & privacy model
└── pyproject.toml # PEP 621 / 735 packaging, ruff, mypy & pytest config
Contributing
Contributions are welcome! Please read CONTRIBUTING.md and the Development & Testing Guide before opening a pull request. In brief:
- Follow Conventional Commits (
feat:,fix:,docs:,test:,chore:) verified withuv run cz check. - Pass all CI quality gates (
ruff format --check .,ruff check .,ruff check --select S src/,mypy src tests,pytest -m reverse -v,pytest -m pii -v, andpytest -v --covwith branch coverage>= 80%). - Keep all Phase 1 and Phase 2 library modules free of config/policy-file reading and external UI/server concerns.
For security vulnerabilities, see SECURITY.md.
Documentation
- Documentation Index — Full guide overview
- Architecture & Internals — Two-phase pipeline flow, intake, crossref resolution, chunking, and transactional staging
- Python API Reference — Signatures, profiles, data shapes, and error handling
- CLI Reference —
docpipe-maestrocommands, TOML config, outputs, and troubleshooting - Retrieval Evaluation Baselines — Offline dense vs
fts5_hybridcomparison metrics (JSON) - Development & Testing Guide — Setup, quality gates, offline test doubles, and Ollama live runs
- Future Features Backlog — Implementation status and post-v0.2 roadmap
- Changelog — Release history
License
This project is licensed under the MIT License.
Metadata
Release files for docpipe-maestro 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| docpipe_maestro-0.2.1.tar.gz | 320.7 kB | With provenance | Details for docpipe_maestro-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| docpipe_maestro-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for docpipe_maestro-0.2.1-py3-none-any.whl |
Total release size: 451.9 kB
Release files / docpipe_maestro-0.2.1.tar.gz
| Download URL | docpipe_maestro-0.2.1.tar.gz |
|---|---|
| Size | 320.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
901c41b9f6e1aa07e4c66c8facdbc6ef771975d90f93a92141f5e54b58cf8faf
|
|
BLAKE2b-256 checksum How to use checksums |
c96bb712ea231c977b5b278a876d47ff18efe40295b43593f916988476c5079b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logRelease files / docpipe_maestro-0.2.1-py3-none-any.whl
| Download URL | docpipe_maestro-0.2.1-py3-none-any.whl |
|---|---|
| Size | 131.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7b69bb08fc2054389cef4c359f2e36d0c3f532f7d41bb9758356412dccb77db3
|
|
BLAKE2b-256 checksum How to use checksums |
c99ad5f0c1d25cc5f319de923d0f16571b12e6fe948c4ec350f418a2d0d5f588
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logSecurity for docpipe-maestro 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for docpipe_maestro-0.2.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.