Skip to main content

High-performance document preparation toolkit for LLM and RAG pipelines.

Project description

preembed

preembed

Clean, chunk, deduplicate, and score documents before embedding.

preembed is a high-performance toolkit that sits between your raw documents and your vector database. It turns messy HTML, markdown, logs, and exports into high-quality, LLM-ready chunks — so your RAG pipeline retrieves better context and hallucinates less.

License: MIT Python 3.10+ Rust

Why

Most RAG pipelines embed documents directly — noise, duplicates, and all. The result: wasted tokens, bloated vector stores, and retrieval that returns boilerplate instead of answers.

preembed fixes the data before it enters the pipeline:

raw documents → clean → chunk → dedupe → score → embed
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        preembed handles this

Install

pip install preembed

For development (requires Rust toolchain):

pip install -e ".[dev]"
maturin develop --release

Quickstart

Pipeline (recommended)

from preembed import Pipeline

result = Pipeline(chunk_size=512, overlap=64).run("path/to/documents/")
result.save_report("report.json")

print(f"{result.stats.total_chunks} chunks, {result.stats.duplicate_ratio:.0%} duplicates removed")

Individual primitives

from preembed import clean_text, chunk_text, dedupe_chunks, score_chunks

text = clean_text("<html><script>tracking()</script><p>Actual content here.</p></html>")
# → "Actual content here."

chunks = chunk_text(text, chunk_size=512, overlap=64)
chunks = dedupe_chunks(chunks)
scored = score_chunks(chunks)
# Each chunk gets a 0-1 quality score + warning labels

Streaming (memory-efficient)

for batch in Pipeline().stream("huge_corpus/", batch_size=50):
    upload_to_vector_db(batch.scored_chunks)

Cross-batch deduplication included. Processes one batch at a time — constant memory regardless of corpus size.

Custom scoring

from preembed import register_scorer

@register_scorer("domain_relevance", weight=0.15)
def domain_scorer(chunk: str) -> float:
    return 1.0 if "patient" in chunk.lower() else 0.3

Blends with the built-in scoring engine. Errors default to neutral (0.5) — won't crash your pipeline.

How it works

Stage What it does
Clean Strip HTML, scripts, styles. Normalize whitespace and unicode.
Chunk Structure-aware splitting. Preserves headings, code blocks, paragraphs. Configurable size + overlap.
Dedupe Exact, normalized, and near-duplicate removal via fingerprint similarity.
Score Quality score (0–1) based on length, density, structure, duplicates. Warning labels for boilerplate, too-short/long, low-density.
Report JSON, Markdown, or HTML diagnostics with per-chunk scores and recommendations.

API

# Core primitives
clean_text(text: str) -> str
chunk_text(text: str, chunk_size=512, overlap=64, mode="word", tokenizer=None) -> list[str]
dedupe_chunks(chunks, return_stats=False, near_duplicate_threshold=0.9) -> list[str] | DedupeResult
score_chunks(chunks) -> list[Chunk]

# Pipeline
Pipeline(chunk_size=512, overlap=64, dedupe=True, score=True)
Pipeline.run(source, progress=None) -> PipelineResult
Pipeline.stream(source, batch_size=50, progress=None) -> Iterator[PipelineResult]

# Extensibility
register_scorer(name, weight=0.1)(fn)
clear_scorers()

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest

License

MIT

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

preembed-0.1.1.tar.gz (26.2 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

preembed-0.1.1-cp312-cp312-win_amd64.whl (182.2 kB view details)

Uploaded CPython 3.12Windows x86-64

preembed-0.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (320.6 kB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

preembed-0.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (313.3 kB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ ARM64

preembed-0.1.1-cp312-cp312-macosx_11_0_arm64.whl (280.9 kB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

preembed-0.1.1-cp311-cp311-win_amd64.whl (181.8 kB view details)

Uploaded CPython 3.11Windows x86-64

preembed-0.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (320.0 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

preembed-0.1.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (313.5 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ ARM64

preembed-0.1.1-cp311-cp311-macosx_11_0_arm64.whl (280.8 kB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

preembed-0.1.1-cp310-cp310-win_amd64.whl (181.9 kB view details)

Uploaded CPython 3.10Windows x86-64

preembed-0.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (320.1 kB view details)

Uploaded CPython 3.10manylinux: glibc 2.17+ x86-64

preembed-0.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (313.6 kB view details)

Uploaded CPython 3.10manylinux: glibc 2.17+ ARM64

preembed-0.1.1-cp310-cp310-macosx_11_0_arm64.whl (280.8 kB view details)

Uploaded CPython 3.10macOS 11.0+ ARM64

File details

Details for the file preembed-0.1.1.tar.gz.

File metadata

  • Download URL: preembed-0.1.1.tar.gz
  • Upload date:
  • Size: 26.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for preembed-0.1.1.tar.gz
Algorithm Hash digest
SHA256 55fb81c47371da0562450fa88af74bb15ac6d6df4566a980048c5d6cb9b573d8
MD5 8792a8890eea74f31cd94c55f26e8f34
BLAKE2b-256 cd6ec4079692c8e7b01a37fe24595efd11b52f4081764b9ffce3f494830a83a0

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1.tar.gz:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp312-cp312-win_amd64.whl.

File metadata

  • Download URL: preembed-0.1.1-cp312-cp312-win_amd64.whl
  • Upload date:
  • Size: 182.2 kB
  • Tags: CPython 3.12, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for preembed-0.1.1-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 a0515994bfc2d2820dea17d8257b14b69ce91c55f12a916b0a0ba99dc7f95700
MD5 904923770062e252771c04241ccd176a
BLAKE2b-256 e93418474ba63c1b09952a45d23554c790172bcff240db84758d0015841cbc1c

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp312-cp312-win_amd64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for preembed-0.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e2b425c21d4f53b7f3d5218921262683ca17ca58258fd9be8b777e972f80c3c0
MD5 32e8aa07ce01ca983d9a3cc1baf39c2f
BLAKE2b-256 a9290b5abe419cd43ba9ae44355d9f01c0b91e3f9a24e58f937d4477af8b5ee3

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for preembed-0.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 41f5b771c97d9e40ada75611a2335430b57b8af99413770f0c85e14f82bb34d6
MD5 fe556c6a77a71bff908cebad5675ddac
BLAKE2b-256 e3341526236dced84b7983fc5c5285a150d7871cf46442cdf8def90ab2884058

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for preembed-0.1.1-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 cc76f90868ec6ffe4adc77d70811fbc48ba494f06519bd1fe7e96ba06cf05daa
MD5 26d049de221095d16720d11963e9bfd9
BLAKE2b-256 5d97133abd7cfe061bbe4a5515e9e07593e41e6d0566a6d725a5235a756a3db8

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: preembed-0.1.1-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 181.8 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for preembed-0.1.1-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 5e12fdd61b69a645fe238c6f17b72a692e2a0766d9d6ff05d07c501577211436
MD5 ecc63e000cbb394c8ff9a16fc8a02b3e
BLAKE2b-256 20429454ee7eae1c59f6aed8417fa7508f56ea3c53f09e723e2da91eb284ac65

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp311-cp311-win_amd64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for preembed-0.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4877aaa34c762407b5bb85103ce0a052947f9a07fc0c6ee72dbb022f2c90864e
MD5 27277c0c58582f71bec44ea761d51f02
BLAKE2b-256 a812d1aec47d22dde248b1195f110ddc5daa833dabbcd970e40df040c7ffd1e7

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for preembed-0.1.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8e0dab9471e6f0de0d7d59c991c0f23d3dbb59627d2708aa6e3f530b38cfd2d6
MD5 6b0539dfd8eff656c77851e2b99a058a
BLAKE2b-256 7aa3ae341dc1fd7e0cd2569a8661cf73c368fae5d3840f4f3eea6896da2e5b3c

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for preembed-0.1.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 570f71ee0d60259b9bbc72f06fd3bb94e3060478a962db51c19e57def9cc993d
MD5 079317fa07200565bf0e4106f5167ba7
BLAKE2b-256 33b80da4667e08193f56c193af4a34f40c9a3a955bf2bdba693004d15ebc8b5a

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: preembed-0.1.1-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 181.9 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for preembed-0.1.1-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 6f82608206db5c05b487c7854de18a8df907d70500a94d6be9dce545a37e1af4
MD5 6d1a016eb8062d1e31b36da29d9ea77c
BLAKE2b-256 4867944610e02adf2076a650121cb3cae219d22ea3e18b68c864808f6738d8e6

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp310-cp310-win_amd64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for preembed-0.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8cd6b1e644b7382996803ffe7859d3ea3f3740a95c3175f486a475d7dbd3a5c9
MD5 7db1d97ee23f6fda272462c8542ddbb1
BLAKE2b-256 66852afad7eb1b01e8b0a4e835f778f8f780b12482f36672dab7ed8f7bd94f90

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for preembed-0.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ccb91a797c6fc073e6205f1827ad3c8525449c72f86b3d819ae1b3bb56c18125
MD5 7b7f0df0a259bb8c7775ec47dee00c4f
BLAKE2b-256 38d56c873612926d9f7a8104b5258a4549478997252c46cc83ec0193011cd8ac

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file preembed-0.1.1-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for preembed-0.1.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 93a346d6048a46a410125e57969114caee623991c0d1fdbd42c733ab5f8ee90f
MD5 1af29f6af64d76fdd4ad3c8d362dac1d
BLAKE2b-256 f520af51dac14ebd6393ddcee3e0c1342db8900f88c35649ff854c6a8557f423

See more details on using hashes here.

Provenance

The following attestation bundles were made for preembed-0.1.1-cp310-cp310-macosx_11_0_arm64.whl:

Publisher: publish.yml on AdityaMunot/preembed

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page