rustchunker 1.0.0

A fast document chunking library for RAG pipelines

⚡ rustchunker

Chunk 100,000 documents in ~13 seconds.

A blazing-fast document chunker for RAG pipelines.
Rust core · Python API · parallel across every CPU core.

---

Splitting documents into chunks is step one of every RAG pipeline. rustchunker does it with a Rust core and real multi-core parallelism — so a corpus that takes LangChain or LlamaIndex minutes takes rustchunker seconds.

Chunking 100,000 real documents	Time
rustchunker ⚡	12.7s
semantic-text-splitter	90.6s
chonkie	108.7s
LangChain	113.5s
LlamaIndex	158.4s

_{12 cores · cc_news corpus · sentence strategy. Full methodology, a five-library comparison, memory, correctness, and the results that don't favour us → BENCHMARKS.md.}

✨ Why rustchunker

• ⚡ Built for scale — ~17–24k chunks/sec, and it pulls further ahead as the corpus grows.
• 🧵 Actually parallel — chunk_files fans your whole corpus across every core with rayon, GIL released. Pure-Python libraries can't (they're stuck paying multiprocessing overhead).
• 🎯 Clean sentence boundaries — the sentence strategy splits only on real Unicode sentence ends: 100% clean boundaries in testing, vs 6–99% for the others. (With overlap > 0, chunk heads gain a few words of leading context that may begin mid-sentence — see Strategies.)
• 🪶 Light — low memory, and zero Python runtime dependencies.
• 🔌 Drop-in — pip install, two lines of code, fully typed.
• 📄 Reads your files — .txt, .md, and .html parsed and stripped for you.

📦 Install

pip install rustchunker

🚀 Quickstart

from rustchunker import chunk, chunk_files

# One string
for c in chunk("Your long document…", max_tokens=256, overlap=20, strategy="sentence"):
    print(c.index, c.text)

# Thousands of files, in parallel across all cores
chunks = chunk_files(
    ["doc1.md", "doc2.txt", "doc3.html"],
    max_tokens=256,
    overlap=20,
    strategy="sentence",
    on_error="skip",   # drop unreadable files instead of aborting the batch
)

Files must be UTF-8 (a leading byte-order mark is stripped for you). By default chunk_files raises on the first file it can't read or parse; pass on_error="skip" to drop failing files and keep the rest — handy when ingesting a large, messy corpus where one bad file shouldn't sink the whole run.

Each Chunk has .text, .start / .end (character offsets into the source document, so source[c.start:c.end] == c.text), .index, and .metadata (source_file, total_chunks, strategy_used, …).

🧠 Strategies

strategy	what it does
"fixed"	every N tokens, exact — the fast baseline
"sentence"	splits on real sentence boundaries (Unicode-aware; keeps "Dr." and "U.S.A." intact), never exceeding max_tokens

overlap shares N trailing tokens between consecutive chunks — respecting word boundaries — so context isn't lost at the seams. With the sentence strategy this leading context is taken at word granularity, so an overlapped chunk's head can begin part-way through a sentence (its interior boundaries stay clean). (More strategies on the way.)

What "token" means here: a token is a whitespace-delimited word, not a model / BPE token (tiktoken, SentencePiece, …). max_tokens caps words per chunk. Word count only approximates an embedder's token budget (roughly ~0.75×), so leave headroom when sizing chunks against a hard model limit.

🎯 When to use it

Great fit: ingesting a corpus — a knowledge base, document store, or crawl of many files. That's where the parallelism pays off, and it's the common RAG ingestion case.

Honest caveat: for a single document, rustchunker is about as fast as LangChain — the win is at corpus scale, not per call. The full breakdown of "how much is the Rust core vs just using all cores" is in BENCHMARKS.md, decomposed and measured.

⚙️ How it works

The chunking runs in Rust via PyO3, and chunk_files processes files in parallel with rayon while releasing the Python GIL — so other threads keep running and you get true multi-core throughput a pure-Python library can't reach without the overhead of spawning processes.

🛠️ Building from source

git clone https://github.com/salik-elbachir/rustchunker
cd rustchunker
maturin develop --release   # requires Rust + maturin; always use --release for real speed
pytest

📄 License

Dual-licensed under MIT or Apache-2.0, at your option.