docprep 0.1.1

Prepare documents into structured, vector-ready data

docprep

Deterministic document chunking for RAG pipelines.

What is docprep?

docprep transforms source documents into structured, vector-ready chunks with deterministic IDs, Markdown-aware boundaries, and incremental sync. It sits between your documents and your vector store:

Source files → Loader → Parser → Chunker(s) → Sink → Export
                                      │
                                Diff Engine → Changed-only export

docprep produces the same chunk IDs for the same input, every time. When documents change, it computes a structural diff and exports only the added, modified, or deleted chunks — so you re-embed only what changed.

What docprep is NOT

• Not a document parser. Use MarkItDown, Docling, or Unstructured for PDFs/DOCX/PPTX, then feed Markdown into docprep via adapters.
• Not an embedding service. docprep produces text chunks; you bring your own embedding model.
• Not a vector database. docprep exports records for Qdrant, pgvector, Chroma, or any other store.
• Not a RAG framework. Use LlamaIndex or LangChain for retrieval. docprep handles the ingestion layer.

How docprep compares

Feature	docprep	MarkItDown	Docling	Unstructured	Chonkie
Deterministic chunk IDs	✅	N/A	❌	❌	❌
Markdown-aware splitting	✅	N/A	Limited	Limited	❌
Incremental sync (diff)	✅	❌	❌	❌	❌
Multi-format parsing	Via adapters	✅	✅	✅	❌
Plugin system	✅	❌	❌	❌	❌
Chunk-level provenance	✅	N/A	Partial	Partial	❌

Installation

pip install docprep

For PostgreSQL support:

pip install docprep[postgres]

Quick Start

Config-first (recommended)

Create a docprep.toml in your project root:

source = "docs/"

[sink]
database_url = "sqlite:///docs.db"
create_tables = true

[[chunkers]]
type = "heading"

[[chunkers]]
type = "token"
max_tokens = 512

Then run:

docprep ingest              # Ingest documents
docprep preview             # Preview structure without persisting
docprep export -o out.jsonl # Export as JSONL
docprep diff                # Show what changed since last ingest

Python API

from docprep import ingest

result = ingest("docs/")
for doc in result.documents:
    print(f"{doc.title}: {len(doc.sections)} sections, {len(doc.chunks)} chunks")

With database persistence

from sqlalchemy import create_engine
from docprep import ingest
from docprep.sinks.sqlalchemy import SQLAlchemySink

engine = create_engine("sqlite:///docs.db")
sink = SQLAlchemySink(engine=engine)

result = ingest("docs/", sink=sink)
print(f"Persisted: {result.persisted}, Skipped: {len(result.skipped_source_uris)}")

Changed-only export

docprep export docs/ --changed-only --db sqlite:///docs.db -o delta.jsonl

Documentation

Guide	Description
Getting Started	Installation, first ingestion, basic usage
Configuration	docprep.toml reference and all options
CLI Reference	All commands, flags, and examples
Python API	Types, functions, and usage patterns
Architecture	Pipeline flow, identity model, module map
Export	VectorRecordV1, JSONL, changed-only export
Plugins	Entry-point plugin system
Adapters	External converter integration

Design decisions are documented as Architecture Decision Records.

Supported Formats

Format	Extensions	Parser	Notes
Markdown	.md	Built-in	Frontmatter extraction, heading hierarchy
Plain text	.txt	Built-in	First non-empty line as title
HTML	.html, .htm	Built-in (stdlib)	Strips script/style, converts headings
reStructuredText	.rst	Built-in	Heading adornments, field lists
Any format	*	Via adapter	MarkItDown, Docling, Unstructured, etc.

Development

git clone https://github.com/yeongseon/docprep.git
cd docprep
make install

make check-all    # lint + typecheck + test + security
make test         # pytest
make lint         # ruff + mypy
make format       # ruff format

See CONTRIBUTING.md for the full development guide.

License

MIT