kitoken
Tokenizer for language models.
Tokenize text for Llama, Gemini, GPT-5, DeepSeek, Mistral and many others; in the web, on the client and any platform.
from kitoken import Kitoken
encoder = Kitoken.from_web("hf:Qwen/Qwen3.5-9B")
tokens = encoder.encode("hello world!", True)
string = encoder.decode(tokens).decode("utf-8")
assert string == "hello world!"
Overview
Kitoken is a fast and versatile tokenizer for language models compatible with SentencePiece, HuggingFace Tokenizers, OpenAI Tiktoken and Mistral Tekken, supporting BPE, Unigram and WordPiece tokenization.
- Fast and efficient tokenization
Faster than most other tokenizers in both common and uncommon scenarios; see the benchmarks for comparisons with different datasets. - Runs in all environments
Native in Rust and with bindings for Web, Node and Python; see kitoken.dev for a web demo. - Supports input and output processing
Including unicode-aware normalization, pre-tokenization and post-processing options. - Compact data encoding
Definitions are stored in an efficient binary format and without merge list.
See the main README for more information.
1 maintainer
Release files for kitoken 0.11.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| kitoken-0.11.0.tar.gz | 70.3 kB | Details |
Built distributions (wheels)
Total release size: 32.7 MB
Release files / kitoken-0.11.0.tar.gz
| Download URL | kitoken-0.11.0.tar.gz |
|---|---|
| Size | 70.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e97b1b91fc92871ea361007eec0bee079ae857d49da0689bdac9d1cc596e9af0
|
|
BLAKE2b-256 checksum How to use checksums |
6c72b8bc85b00eb0739007b4c3e41556e31f104ff126581eeb838824d7ee6a6b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-win_amd64.whl
| Download URL | kitoken-0.11.0-cp310-abi3-win_amd64.whl |
|---|---|
| Size | 2.4 MB |
| Tags | CPython 3.10 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
5ddc90bf254980822895f1f6437fd60648d4abbf4361a3e532e1c6d0a4a62611
|
|
BLAKE2b-256 checksum How to use checksums |
6166cf7aef64526ddfaf3f44e531cfd3492617149e23aa13fe991f7b39b695ff
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-win32.whl
| Download URL | kitoken-0.11.0-cp310-abi3-win32.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.10 Windows x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
a03fb61dc26961b6377f8f57091b30ca191922fe4d9148845a2ad016b71ffe0f
|
|
BLAKE2b-256 checksum How to use checksums |
174fe21873bcb7a1b37cd113eec0304ac01c7f83f40408dfa5733aed7375d0cf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-musllinux_1_2_x86_64.whl
| Download URL | kitoken-0.11.0-cp310-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 2.8 MB |
| Tags | CPython 3.10 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
52e65b489a5a75b86db02a14eb06783667525ba086be9020c296d8ea9ec84138
|
|
BLAKE2b-256 checksum How to use checksums |
0d090dd25708e252af51c027e89b6adb51858d5b20ca700039c808291cce051a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-musllinux_1_2_i686.whl
| Download URL | kitoken-0.11.0-cp310-abi3-musllinux_1_2_i686.whl |
|---|---|
| Size | 2.4 MB |
| Tags | CPython 3.10 Linux musl 1.2+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
cb8be1fc5600e2c55157ac82d2b8405a72a743f4a5b0ab6d29f7ced848d98756
|
|
BLAKE2b-256 checksum How to use checksums |
a5a55a63ff6203a47f932111bfb01bf6555527fdfdfd733e967e0280e2963908
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-musllinux_1_2_armv7l.whl
| Download URL | kitoken-0.11.0-cp310-abi3-musllinux_1_2_armv7l.whl |
|---|---|
| Size | 2.3 MB |
| Tags | CPython 3.10 Linux musl 1.2+ ARMv7l abi3 |
|
SHA-256 checksum How to use checksums |
2800c229596b9a3cad8cea000db827d9ec1ce6052f30b7e656957f50740076df
|
|
BLAKE2b-256 checksum How to use checksums |
2a8d1cae0a1eb40a2744dcfe0e00d7b184e069d55d21cba0bf636c371dd0dede
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-musllinux_1_2_aarch64.whl
| Download URL | kitoken-0.11.0-cp310-abi3-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 2.7 MB |
| Tags | CPython 3.10 Linux musl 1.2+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
606a54e37cf8c8f139ff4c19a2968f387718c15c1f2a41b54eac66db45264911
|
|
BLAKE2b-256 checksum How to use checksums |
9feedd511284df08f26ccb97ff38e80c289f3ba27c6a9145877e344a5d6ea414
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-manylinux_2_28_x86_64.whl
| Download URL | kitoken-0.11.0-cp310-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 2.8 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
8b6a0ab17163e569abab9ef5e4deae843a213130b0e05db5d793d8248969b4f3
|
|
BLAKE2b-256 checksum How to use checksums |
b2ba96549d7cbf2ac8befdb595149679d2b6f75122efbf045fcc49a22a8477d0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-manylinux_2_28_ppc64le.whl
| Download URL | kitoken-0.11.0-cp310-abi3-manylinux_2_28_ppc64le.whl |
|---|---|
| Size | 2.5 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ PowerPC 64-le abi3 |
|
SHA-256 checksum How to use checksums |
2e108b8616cd756be536340d50843dec8c294a82c0db7d0ea4cfffd258f5cb9d
|
|
BLAKE2b-256 checksum How to use checksums |
398e0c97951ae60059ecc2ad394c82b54901299f944ea29366816be19e683c7f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-manylinux_2_28_i686.whl
| Download URL | kitoken-0.11.0-cp310-abi3-manylinux_2_28_i686.whl |
|---|---|
| Size | 2.5 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
4a0eea6b9c4a58422ce06659d8e07c1d67e08f7fb5d31de5d84e89bcc28ca8f3
|
|
BLAKE2b-256 checksum How to use checksums |
3576cd776711bb0522735987e642dbe19a9148df22c758ccbb8580d961501dbc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-manylinux_2_28_armv7l.whl
| Download URL | kitoken-0.11.0-cp310-abi3-manylinux_2_28_armv7l.whl |
|---|---|
| Size | 2.3 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARMv7l abi3 |
|
SHA-256 checksum How to use checksums |
04129b1236145209f65acee7efefec659a225791bc0412d28582285ee780ec7c
|
|
BLAKE2b-256 checksum How to use checksums |
75f204a3195aeb2316b1a2eab96e9c45c47a857b67ea87914931c2598feacd1c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-manylinux_2_28_aarch64.whl
| Download URL | kitoken-0.11.0-cp310-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 2.7 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
a19dc3f943ba5d3f16a75219a28bc1d916578cf7262099950102b12ef726d195
|
|
BLAKE2b-256 checksum How to use checksums |
baf16d5fa5a95e8752c6e7463d79d78b3093bfca02ed5129f122081b9b019c93
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-macosx_13_0_x86_64.whl
| Download URL | kitoken-0.11.0-cp310-abi3-macosx_13_0_x86_64.whl |
|---|---|
| Size | 2.7 MB |
| Tags | CPython 3.10 abi3 macOS 13.0+ x86-64 |
|
SHA-256 checksum How to use checksums |
fc8ac6642f22464fdb83524a052054706670256537ea71c2675902f94cdb533b
|
|
BLAKE2b-256 checksum How to use checksums |
c213ed509af8c8d61241cbb6a0c171431295d0bc7204e93e878b72cb6bf37d5f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
Release files / kitoken-0.11.0-cp310-abi3-macosx_13_0_arm64.whl
| Download URL | kitoken-0.11.0-cp310-abi3-macosx_13_0_arm64.whl |
|---|---|
| Size | 2.5 MB |
| Tags | CPython 3.10 abi3 macOS 13.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
25a3894f63eb4d8c0229a58310d40e19aa3d615afb45717b015f5cb938ea4572
|
|
BLAKE2b-256 checksum How to use checksums |
0bf2e1790393c6b1fa5b1a7a9470f498aa03b8a2a3ef74492c96e37dcec9cd21
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.13.2
|
