docprep

Deterministic document chunking for RAG pipelines.

What is docprep?

docprep transforms source documents into structured, vector-ready chunks with deterministic IDs, Markdown-aware boundaries, and incremental sync. It sits between your documents and your vector store:

Source files → Loader → Parser → Chunker(s) → Sink → Export
                                      │
                                Diff Engine → Changed-only export

docprep produces the same chunk IDs for the same input, every time. When documents change, it computes a structural diff and exports only the added, modified, or deleted chunks — so you re-embed only what changed.

What docprep is NOT

• Not a document parser. Use MarkItDown, Docling, or Unstructured for PDFs/DOCX/PPTX, then feed Markdown into docprep via adapters.
• Not an embedding service. docprep produces text chunks; you bring your own embedding model.
• Not a vector database. docprep exports records for Qdrant, pgvector, Chroma, or any other store.
• Not a RAG framework. Use LlamaIndex or LangChain for retrieval. docprep handles the ingestion layer.

How docprep compares

Installation

pip install docprep

For PostgreSQL support:

pip install docprep[postgres]

Quick Start

Config-first (recommended)

Create a docprep.toml in your project root:

source = "docs/"

[sink]
database_url = "sqlite:///docs.db"
create_tables = true

[[chunkers]]
type = "heading"

[[chunkers]]
type = "token"
max_tokens = 512

Then run:

docprep ingest              # Ingest documents
docprep preview             # Preview structure without persisting
docprep export -o out.jsonl # Export as JSONL
docprep diff                # Show what changed since last ingest

Python API

from docprep import ingest

result = ingest("docs/")
for doc in result.documents:
    print(f"{doc.title}: {len(doc.sections)} sections, {len(doc.chunks)} chunks")

With database persistence

from sqlalchemy import create_engine
from docprep import ingest
from docprep.sinks.sqlalchemy import SQLAlchemySink

engine = create_engine("sqlite:///docs.db")
sink = SQLAlchemySink(engine=engine)

result = ingest("docs/", sink=sink)
print(f"Persisted: {result.persisted}, Skipped: {len(result.skipped_source_uris)}")

Changed-only export

docprep export docs/ --changed-only --db sqlite:///docs.db -o delta.jsonl

Documentation

Design decisions are documented as Architecture Decision Records.

Supported Formats

Development

git clone https://github.com/yeongseon/docprep.git
cd docprep
make install

make check-all    # lint + typecheck + test + security
make test         # pytest
make lint         # ruff + mypy
make format       # ruff format

See CONTRIBUTING.md for the full development guide.

License

MIT