High-performance document preparation toolkit for LLM and RAG pipelines.
Project description
preembed
Clean, chunk, deduplicate, and score documents before embedding.
preembed is a high-performance toolkit that sits between your raw documents and your vector database. It turns messy HTML, markdown, logs, and exports into high-quality, LLM-ready chunks — so your RAG pipeline retrieves better context and hallucinates less.
Why
Most RAG pipelines embed documents directly — noise, duplicates, and all. The result: wasted tokens, bloated vector stores, and retrieval that returns boilerplate instead of answers.
preembed fixes the data before it enters the pipeline:
raw documents → clean → chunk → dedupe → score → embed
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
preembed handles this
Install
pip install preembed
For development (requires Rust toolchain):
pip install -e ".[dev]"
maturin develop --release
Quickstart
Pipeline (recommended)
from preembed import Pipeline
result = Pipeline(chunk_size=512, overlap=64).run("docs/")
result.save_report("report.json")
print(f"{result.stats.total_chunks} chunks, {result.stats.duplicate_ratio:.0%} duplicates removed")
Individual primitives
from preembed import clean_text, chunk_text, dedupe_chunks, score_chunks
text = clean_text("<html><script>tracking()</script><p>Actual content here.</p></html>")
# → "Actual content here."
chunks = chunk_text(text, chunk_size=512, overlap=64)
chunks = dedupe_chunks(chunks)
scored = score_chunks(chunks)
# Each chunk gets a 0-1 quality score + warning labels
Streaming (memory-efficient)
for batch in Pipeline().stream("huge_corpus/", batch_size=50):
upload_to_vector_db(batch.scored_chunks)
Cross-batch deduplication included. Processes one batch at a time — constant memory regardless of corpus size.
Custom scoring
from preembed import register_scorer
@register_scorer("domain_relevance", weight=0.15)
def domain_scorer(chunk: str) -> float:
return 1.0 if "patient" in chunk.lower() else 0.3
Blends with the built-in scoring engine. Errors default to neutral (0.5) — won't crash your pipeline.
Performance
Benchmarked on a 15 MB real-world corpus (Wikipedia, RFCs, Gutenberg, GitHub, MDN, Hacker News):
| Operation | preembed | BeautifulSoup | LangChain | LlamaIndex |
|---|---|---|---|---|
| Cleaning | 110 ms | 1,643 ms | — | — |
| Chunking | 31 ms | — | 1,467 ms | 1,465 ms |
| Full pipeline | 543 ms | — | — | — |
Cleaning is native-compiled. Deduplication uses an inverted fingerprint index — O(1) for exact/normalized matches.
More benchmark comparisons
| Chunker | Time | Chunks | Quality |
|---|---|---|---|
| preembed | 31 ms | 8,136 | 1.000 |
| Chonkie | 103 ms | 8,994 | 0.975 |
| semantic-text-splitter | 211 ms | 8,710 | 0.956 |
| semchunk | 214 ms | 8,396 | 0.967 |
| LangChain Recursive | 1,467 ms | 16,889 | 0.672 |
| LlamaIndex Sentence | 1,465 ms | 16,889 | 0.672 |
Quality scores are heuristic signals (length, density, structure, duplicates), not human relevance judgments. See docs/BENCHMARKS.md for methodology and caveats.
How it works
| Stage | What it does |
|---|---|
| Clean | Strip HTML, scripts, styles. Normalize whitespace and unicode. |
| Chunk | Structure-aware splitting. Preserves headings, code blocks, paragraphs. Configurable size + overlap. |
| Dedupe | Exact, normalized, and near-duplicate removal via fingerprint similarity. |
| Score | Quality score (0–1) based on length, density, structure, duplicates. Warning labels for boilerplate, too-short/long, low-density. |
| Report | JSON, Markdown, or HTML diagnostics with per-chunk scores and recommendations. |
API
# Core primitives
clean_text(text: str) -> str
chunk_text(text: str, chunk_size=512, overlap=64, mode="word", tokenizer=None) -> list[str]
dedupe_chunks(chunks, return_stats=False, near_duplicate_threshold=0.9) -> list[str] | DedupeResult
score_chunks(chunks) -> list[Chunk]
# Pipeline
Pipeline(chunk_size=512, overlap=64, dedupe=True, score=True)
Pipeline.run(source, progress=None) -> PipelineResult
Pipeline.stream(source, batch_size=50, progress=None) -> Iterator[PipelineResult]
# Extensibility
register_scorer(name, weight=0.1)(fn)
clear_scorers()
Development
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
License
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file preembed-0.1.0-cp312-cp312-win_amd64.whl.
File metadata
- Download URL: preembed-0.1.0-cp312-cp312-win_amd64.whl
- Upload date:
- Size: 169.1 kB
- Tags: CPython 3.12, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
668eb35ee02ef681f076e942472322fef5034e1ba5b281c1cb4706560fdcf394
|
|
| MD5 |
bc486fa44b15cdbd6e4b3c3d78a36aae
|
|
| BLAKE2b-256 |
e7bb402cc58e6a5f7fe7ef28c2f4ff469dc23d6fd2043d4975c30ba2c3163be0
|
File details
Details for the file preembed-0.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: preembed-0.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 306.1 kB
- Tags: CPython 3.12, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
437cd753ce4e127b5ceff4fc2629f2829658820350f620582cab9e1a1ec06d50
|
|
| MD5 |
49d33c58bdb63a52ceddb135333fae78
|
|
| BLAKE2b-256 |
b9daf625a8948b9bda312608fa4dcc99cb8a248230d5b9b4ed91b78079ad43b6
|
File details
Details for the file preembed-0.1.0-cp312-cp312-macosx_11_0_arm64.whl.
File metadata
- Download URL: preembed-0.1.0-cp312-cp312-macosx_11_0_arm64.whl
- Upload date:
- Size: 267.6 kB
- Tags: CPython 3.12, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
13844401d6bd13cefdf127b835fad7c1419d0eb95d3389238c86f624e296fd11
|
|
| MD5 |
4e6b933e62886401a43fcc108d6ae55f
|
|
| BLAKE2b-256 |
bc1c09c1b0a3cd946bcbea6e25b8b0cb32a39189eb7e4a02bfab86178a2de79d
|
File details
Details for the file preembed-0.1.0-cp311-cp311-win_amd64.whl.
File metadata
- Download URL: preembed-0.1.0-cp311-cp311-win_amd64.whl
- Upload date:
- Size: 168.6 kB
- Tags: CPython 3.11, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c64f2bfe735affd1ac4e9d7a113fa9775525ed3070937f66e5764321ff6a1965
|
|
| MD5 |
7f5a3642ba47ac4d81e19d1b8a6bd4c5
|
|
| BLAKE2b-256 |
5465e7153826415428c396894576cf5936d153221f192541b3f6d3091ddf6850
|
File details
Details for the file preembed-0.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: preembed-0.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 306.3 kB
- Tags: CPython 3.11, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
90158a37cbc3bf12e4ff2ce7c8280008163efbc05673758317919bc2d59bb743
|
|
| MD5 |
68c65631c87501f10b20f58fa7c8f6ed
|
|
| BLAKE2b-256 |
29874ac8803eea6a389dce26f8ffe4a975e8e55e93b41bea6f00c8e676136cef
|
File details
Details for the file preembed-0.1.0-cp311-cp311-macosx_11_0_arm64.whl.
File metadata
- Download URL: preembed-0.1.0-cp311-cp311-macosx_11_0_arm64.whl
- Upload date:
- Size: 267.8 kB
- Tags: CPython 3.11, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7d720a1a0492b05e6f7b01896d8a3e61c12efc6686fda72aff486eea3b3809f7
|
|
| MD5 |
a1e15d05d6d9b4fdc85a71211677cc2f
|
|
| BLAKE2b-256 |
262f1f3d2160a22fd1e8f55924f5cc2dfb3be61e136997d6404b96e4962ecc67
|
File details
Details for the file preembed-0.1.0-cp310-cp310-win_amd64.whl.
File metadata
- Download URL: preembed-0.1.0-cp310-cp310-win_amd64.whl
- Upload date:
- Size: 168.6 kB
- Tags: CPython 3.10, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9293b92ac7fac0040ce885abcb434ad00b3fddc894324d63135f9d8b2b5b8cf2
|
|
| MD5 |
421ad01db6d3ac47ae3d9ef6ac00695c
|
|
| BLAKE2b-256 |
56b8605ac0d9f11fb4275c4cb2b4758deff2c0e3bd77b5a975fe17c46688f463
|
File details
Details for the file preembed-0.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: preembed-0.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 306.2 kB
- Tags: CPython 3.10, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
374138d45ea2a153e24d4770512af6b2c6e892629a705b3826ea20028b0e8397
|
|
| MD5 |
a5336aadf012ab4b846e09d0569d9c80
|
|
| BLAKE2b-256 |
a82590bade9b947b6719f95c024de36ab88a686fee94c5d104c13d9ee5c23945
|
File details
Details for the file preembed-0.1.0-cp310-cp310-macosx_11_0_arm64.whl.
File metadata
- Download URL: preembed-0.1.0-cp310-cp310-macosx_11_0_arm64.whl
- Upload date:
- Size: 267.9 kB
- Tags: CPython 3.10, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
510f11177bcefec2249810f9700804f4afc088dfc6f67f29bb5829009e7d1815
|
|
| MD5 |
09e8611431b8a86a47e2878c3161324b
|
|
| BLAKE2b-256 |
26d5ba6987abbd410e83d7db96f01a8f9a94343695433ef5e100af83892d5bf5
|
