memchunk 0.4.0

The fastest semantic text chunking library

memchunk

the fastest text chunking library — up to 1 TB/s throughput

---

you know how every chunking library claims to be fast? yeah, we actually meant it.

memchunk splits text at semantic boundaries (periods, newlines, the usual suspects) and does it stupid fast. we're talking "chunk the entire english wikipedia in 120ms" fast.

want to know how? read the blog post where we nerd out about SIMD instructions and lookup tables.

📦 installation

pip install memchunk

looking for rust or javascript?

🚀 usage

from memchunk import Chunker

text = "Hello world. How are you? I'm fine.\nThanks for asking."

# with defaults (4KB chunks, split at \n . ?)
for chunk in Chunker(text):
    print(bytes(chunk))

# with custom size
for chunk in Chunker(text, size=1024):
    print(bytes(chunk))

# with custom delimiters
for chunk in Chunker(text, delimiters=".?!\n"):
    print(bytes(chunk))

# with multi-byte pattern (e.g., metaspace ▁ for SentencePiece tokenizers)
for chunk in Chunker(text, pattern="▁", prefix=True):
    print(bytes(chunk))

# with consecutive pattern handling (split at START of runs, not middle)
for chunk in Chunker("word   next", pattern=" ", consecutive=True):
    print(bytes(chunk))

# with forward fallback (search forward if no pattern in backward window)
for chunk in Chunker(text, pattern=" ", forward_fallback=True):
    print(bytes(chunk))

# collect all chunks
chunks = list(Chunker(text))

chunks are returned as memoryview objects (zero-copy slices of the original text).

📝 citation

if you use memchunk in your research, please cite it as follows:

@software{memchunk2025,
  author = {Minhas, Bhavnick},
  title = {memchunk: The fastest text chunking library},
  year = {2025},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/chonkie-inc/memchunk}},
}

📄 license

licensed under either of Apache License, Version 2.0 or MIT license at your option.