chonkie-core
the fastest text chunking library — up to 1 TB/s throughput
you know how every chunking library claims to be fast? yeah, we actually meant it.
chonkie-core splits text at semantic boundaries (periods, newlines, the usual suspects) and does it stupid fast. we're talking "chunk the entire english wikipedia in 120ms" fast.
want to know how? read the blog post where we nerd out about SIMD instructions and lookup tables.
📦 installation
pip install chonkie-core
looking for rust or javascript?
🚀 usage
from chonkie_core import Chunker
text = "Hello world. How are you? I'm fine.\nThanks for asking."
# with defaults (4KB chunks, split at \n . ?)
for chunk in Chunker(text):
print(bytes(chunk))
# with custom size
for chunk in Chunker(text, size=1024):
print(bytes(chunk))
# with custom delimiters
for chunk in Chunker(text, delimiters=".?!\n"):
print(bytes(chunk))
# with multi-byte pattern (e.g., metaspace ▁ for SentencePiece tokenizers)
for chunk in Chunker(text, pattern="▁", prefix=True):
print(bytes(chunk))
# with consecutive pattern handling (split at START of runs, not middle)
for chunk in Chunker("word next", pattern=" ", consecutive=True):
print(bytes(chunk))
# with forward fallback (search forward if no pattern in backward window)
for chunk in Chunker(text, pattern=" ", forward_fallback=True):
print(bytes(chunk))
# collect all chunks
chunks = list(Chunker(text))
chunks are returned as memoryview objects (zero-copy slices of the original text).
📝 citation
if you use chonkie-core in your research, please cite it as follows:
@software{chunk2025,
author = {Minhas, Bhavnick},
title = {chunk: The fastest text chunking library},
year = {2025},
publisher = {GitHub},
howpublished = {\url{https://github.com/chonkie-inc/chunk}},
}
📄 license
licensed under either of Apache License, Version 2.0 or MIT license at your option.
1 maintainer
Metadata
1 maintainer
Release files for chonkie-core 0.10.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| chonkie_core-0.10.2.tar.gz | 70.0 kB | Details |
Built distributions (wheels)
Total release size: 8.6 MB
Release files / chonkie_core-0.10.2.tar.gz
| Download URL | chonkie_core-0.10.2.tar.gz |
|---|---|
| Size | 70.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c8e40ef8f3a034a7c5dd23a0401dce2ef2b4883f5a6a29cf94176d64b209bdbb
|
|
BLAKE2b-256 checksum How to use checksums |
2017ad31bbcfe1a7b63f76e065684a8e9cd98552d3c627fb28d8fbc05f1a9456
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp314-cp314-win_amd64.whl
| Download URL | chonkie_core-0.10.2-cp314-cp314-win_amd64.whl |
|---|---|
| Size | 228.7 kB |
| Tags | CPython 3.14 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
20c9d64b9c5169d1f7e5ceeaa22bff3613017b8937c224ba85e3e24748fb0ba3
|
|
BLAKE2b-256 checksum How to use checksums |
b093e3167b1823f919f1a4c517092f73dc4107737de8bb53c67ff60c964b5244
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp314-cp314-pyemscripten_2026_0_wasm32.whl
| Download URL | chonkie_core-0.10.2-cp314-cp314-pyemscripten_2026_0_wasm32.whl |
|---|---|
| Size | 101.8 kB |
| Tags | CPython 3.14 PyEmscripten 2026.0+ WebAssembly |
|
SHA-256 checksum How to use checksums |
acf83c774d5646dd9f16b896722155db8e87948fc93ee952fd649b5344d9205d
|
|
BLAKE2b-256 checksum How to use checksums |
f5d0103da313f8990607ac77c4735f372734a4d96afe79fdabc5883731bb2063
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp314-cp314-manylinux_2_28_aarch64.whl
| Download URL | chonkie_core-0.10.2-cp314-cp314-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 370.6 kB |
| Tags | CPython 3.14 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
d1999d993df8ecf941361b8eff9618f2a9bf3436b4a1851ec3f3f42d9a9c1a7e
|
|
BLAKE2b-256 checksum How to use checksums |
781e4c775aec3d7ef6ff6ad90570c5b6f612fda9bf1ae7dfb4bd322d0fe55ffe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | chonkie_core-0.10.2-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 386.2 kB |
| Tags | CPython 3.14 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
e4b6e2cf0411cb675d8b22f2b234b07e20724ded6cf84b2bc625b677885d2ef7
|
|
BLAKE2b-256 checksum How to use checksums |
5605b3e206f063e6515a1b2e1f873ff33d91bf1eaab99554577c47804ebbbf37
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp314-cp314-macosx_11_0_arm64.whl
| Download URL | chonkie_core-0.10.2-cp314-cp314-macosx_11_0_arm64.whl |
|---|---|
| Size | 336.5 kB |
| Tags | CPython 3.14 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
5266bdb8887c223782096c97a72eb49f19930f797213fa65b82199042742dc98
|
|
BLAKE2b-256 checksum How to use checksums |
264d29e42ac094624a0ebfe74684ce62eb870e640679c1feb8da63e631a45cde
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp314-cp314-macosx_10_12_x86_64.whl
| Download URL | chonkie_core-0.10.2-cp314-cp314-macosx_10_12_x86_64.whl |
|---|---|
| Size | 346.8 kB |
| Tags | CPython 3.14 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
cb0626fa4d9df5128188eef069f69809571fbfc6db47e348d5f4d1199fa34b9c
|
|
BLAKE2b-256 checksum How to use checksums |
0b45e4d68840847133afa9c69fc0e28575270afdd9dcc0c13280a9b40445377d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp313-cp313-win_amd64.whl
| Download URL | chonkie_core-0.10.2-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 229.4 kB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
9f77811bb722bd019a52353364bc6dcea1a9998120a413f1c2a1c79153a66386
|
|
BLAKE2b-256 checksum How to use checksums |
3839158f85b728d84e85a00768ccebe3a289d21462f52ba7850db9e01e61e4df
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp313-cp313-manylinux_2_28_aarch64.whl
| Download URL | chonkie_core-0.10.2-cp313-cp313-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 371.7 kB |
| Tags | CPython 3.13 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
72974163eab058d14c365f45b53d2367b1e04a0dcd779dedd828973cf21256d7
|
|
BLAKE2b-256 checksum How to use checksums |
046e2c505276878b695b4cee07eb35a5c536ed92d60c722b4547da1517fb3e41
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | chonkie_core-0.10.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 386.4 kB |
| Tags | CPython 3.13 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
aebeba2d2cb0d30eefdbdb18d2ab2fcfbc0ea0d5121f61f537a6114a1da56101
|
|
BLAKE2b-256 checksum How to use checksums |
e976079296e83d2ed6e34b8ed24bd23e59193b502b53ddfa176b3e921410a5db
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp313-cp313-macosx_11_0_arm64.whl
| Download URL | chonkie_core-0.10.2-cp313-cp313-macosx_11_0_arm64.whl |
|---|---|
| Size | 336.6 kB |
| Tags | CPython 3.13 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
3da60e797b2fcbef3e17fad3e46bb7d4f5541ac5f87a5556c2c610cef83a549a
|
|
BLAKE2b-256 checksum How to use checksums |
3f33b640f406b6d2fe30d5c41d4a91411d9ed1dc22acf6497cda837a6f121ba4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp313-cp313-macosx_10_12_x86_64.whl
| Download URL | chonkie_core-0.10.2-cp313-cp313-macosx_10_12_x86_64.whl |
|---|---|
| Size | 346.8 kB |
| Tags | CPython 3.13 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
25372879e43b235dee01f48309634a3fce07b55c382a7fbe200d6b66769267e4
|
|
BLAKE2b-256 checksum How to use checksums |
594f0814d2bd9abddbbc37f8c204f056d28f2ab7d14bf742515e9684062bd758
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp312-cp312-win_amd64.whl
| Download URL | chonkie_core-0.10.2-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 229.7 kB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
ac686d192c4dd7e038cea4aba59c677d45e2b1e8de8eb1dd45d269b22a4ae201
|
|
BLAKE2b-256 checksum How to use checksums |
8cc205c2c85b11d092f04768c41fb9cfdd27a9d27fa7e10e107188a51ce8813f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp312-cp312-manylinux_2_28_aarch64.whl
| Download URL | chonkie_core-0.10.2-cp312-cp312-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 371.5 kB |
| Tags | CPython 3.12 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
5a09bcb56d9d9f2fa6ec84485aeecbcaa375567eea9fbec18517d2ef643b9029
|
|
BLAKE2b-256 checksum How to use checksums |
89c8b0d91419730bff4566305f2c85f27fb1378e8e1e99f8e59403c6b25d43a2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | chonkie_core-0.10.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 387.2 kB |
| Tags | CPython 3.12 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
51dbf2163dac5ad695dde3f92155c6febb9332b5fb5a174ac80ae9fc8f8ccde9
|
|
BLAKE2b-256 checksum How to use checksums |
0d9fcbfec0f25e35d2a5b62a9cba7a5a85cfc5349210b9f9acb40832060211e4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp312-cp312-macosx_11_0_arm64.whl
| Download URL | chonkie_core-0.10.2-cp312-cp312-macosx_11_0_arm64.whl |
|---|---|
| Size | 336.9 kB |
| Tags | CPython 3.12 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
a6d081363fbfaad0f36c66e67e6dec1c9f9850c04c0475b64b3e47084257b646
|
|
BLAKE2b-256 checksum How to use checksums |
b23fba4083f21b4ef52ed130a79f371fed453b7edf998bf08079694137880bff
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp312-cp312-macosx_10_12_x86_64.whl
| Download URL | chonkie_core-0.10.2-cp312-cp312-macosx_10_12_x86_64.whl |
|---|---|
| Size | 347.5 kB |
| Tags | CPython 3.12 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
31f8fa3415d2e93cc3bdb1e99c3b6865278169af348c68fb2943aed2fcabc010
|
|
BLAKE2b-256 checksum How to use checksums |
562f4dd88b4af9ef0e9ed31a3c6a3d8e44327c5ca22b074076830b0224e0ad58
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp311-cp311-win_amd64.whl
| Download URL | chonkie_core-0.10.2-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 231.0 kB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
cbe5a8a1e89a79a74bac99409e1bf5e7a5b3e2286ac5f2185077d39f5fa95175
|
|
BLAKE2b-256 checksum How to use checksums |
5462425bca737db62a371b1e3393d76f69f483baf3482ccd88504089501ee0e4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp311-cp311-manylinux_2_28_aarch64.whl
| Download URL | chonkie_core-0.10.2-cp311-cp311-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 374.6 kB |
| Tags | CPython 3.11 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
f69969fa3f75930bd10f1ec502993b758df4e5a62d170c373a4fa73743dc2f93
|
|
BLAKE2b-256 checksum How to use checksums |
50654a86e1648147df94bcc2d681b6c5f40a21c57df65b45bf2f3ceb72fcb784
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | chonkie_core-0.10.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 390.0 kB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
ff79cbe5016f85a8fac2164e1716f3335b631704bfa486f0d1de02c6d88c2bce
|
|
BLAKE2b-256 checksum How to use checksums |
a505513baf0c159fb93e6c05bf199cca79045b2c186e9fea1fd7ebf47d063c7b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp311-cp311-macosx_11_0_arm64.whl
| Download URL | chonkie_core-0.10.2-cp311-cp311-macosx_11_0_arm64.whl |
|---|---|
| Size | 339.7 kB |
| Tags | CPython 3.11 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
712ecc448302bdc8f3ab4aab1e17d5d37b50d57d89d4fa4ec44821a37eb8fa57
|
|
BLAKE2b-256 checksum How to use checksums |
93a6069b46520a23264585ddacd117bee62594afff0f851cd1dfedf32b900faf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp311-cp311-macosx_10_12_x86_64.whl
| Download URL | chonkie_core-0.10.2-cp311-cp311-macosx_10_12_x86_64.whl |
|---|---|
| Size | 349.5 kB |
| Tags | CPython 3.11 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
a431af90ac4e4072e02699ac1300efcaaff2ee432b052fe51027998859af1055
|
|
BLAKE2b-256 checksum How to use checksums |
c90b4c28270b24d9e756c90a52966f26ddff5fa2f639aed7e6cd9cc1e1728176
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp310-cp310-win_amd64.whl
| Download URL | chonkie_core-0.10.2-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 230.9 kB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
c7c2c431d86fbae7c1f8d0b8628236df27b4dab040dc6504ded209be9e36f5b0
|
|
BLAKE2b-256 checksum How to use checksums |
16083f55dbb5cd033b9c33e9e5fbaa5ee88af3dbc78e6858cf42341c903c5cc6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp310-cp310-manylinux_2_28_aarch64.whl
| Download URL | chonkie_core-0.10.2-cp310-cp310-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 375.2 kB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
e7e05701463e8e7a339a7be617fe2306153dcb5f269f43251e1516f40ce91134
|
|
BLAKE2b-256 checksum How to use checksums |
0962c3ae371065d31e52eb96d05ef02384521c99dd6bd036bc4b1d7a09ea4cbb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | chonkie_core-0.10.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 390.1 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
2225d6e6622b26d4542ed9661e71b51dfd8b4c5ae7e652fe85b4f0faa4be8eba
|
|
BLAKE2b-256 checksum How to use checksums |
12bd373ca97e84a4c4caf9adba9e710bc48f50c87524379a949cd17ecc588def
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp310-cp310-macosx_11_0_arm64.whl
| Download URL | chonkie_core-0.10.2-cp310-cp310-macosx_11_0_arm64.whl |
|---|---|
| Size | 339.2 kB |
| Tags | CPython 3.10 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
86c3e04603ce69d380f3f6a8f14f85dd902410ad1ec86709ddcfad43dc968946
|
|
BLAKE2b-256 checksum How to use checksums |
ed9d2ed7af6666a168fcb18328e06354035f506aa8965a479827aa6f5528b189
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / chonkie_core-0.10.2-cp310-cp310-macosx_10_12_x86_64.whl
| Download URL | chonkie_core-0.10.2-cp310-cp310-macosx_10_12_x86_64.whl |
|---|---|
| Size | 349.2 kB |
| Tags | CPython 3.10 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
b63d25a3270a98eb5a6739072315f36566c87139078797a0970bc8d480f5bbe0
|
|
BLAKE2b-256 checksum How to use checksums |
9ea016ff6f6d3ddfb3f39d89663717e91eaa1bd5ba6ac52ec2fa01795ac33c65
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
