Usage Guide for FlashTokenizer
FlashTokenizer is a high-performance tokenizer implemented in C++ for efficient LLM inference. It's designed to be significantly faster and equally accurate compared to traditional tokenizers.
Installation
Install FlashTokenizer easily using pip:
pip install -U flash-tokenizer
Or from source:
git clone https://github.com/NLPOptimize/flash-tokenizer
cd flash-tokenizer/prj
pip install .
Prerequisites
- Windows (AMD64), MacOS (ARM64), Ubuntu (x86-64)
- Python 3.8 to 3.13
- g++, clang++, or MSVC
Quick Start
FlashTokenizer supports various pretrained models:
from flash_tokenizer import BertTokenizerFlash
print(*BertTokenizerFlash.get_pretrained(), sep="\n")
Output:
bert-base-cased
bert-base-uncased
bert-base-chinese
bert-base-multilingual-cased
bert-base-multilingual-uncased
kcbert-base
llmlingua-2-bert-base-multilingual-cased-meetingbank
Tokenizing Text
FlashTokenizer usage aligns closely with Hugging Face's BertTokenizer:
from flash_tokenizer import BertTokenizerFlash
from transformers import BertTokenizer
titles = [
'is there any doubt about it "None whatsoever"',
"세상 어떤 짐승이 이를 드러내고 사냥을 해? 약한 짐승이나 몸을 부풀리지, 진짜 짐승은 누구보다 침착하지.",
'そのように二番目に死を偽装して生き残るようになったイタドリがどうして初めて見る自分をこんなに気遣ってくれるのかと尋ねると「私が大切にする人たちがあなたを大切にするから」と答えては'
]
# Load tokenizer with vocab file
tokenizer = BertTokenizerFlash('vocab.txt', do_lower_case=False, model_max_length=512)
for title in titles:
tokens = tokenizer.tokenize(title)
token_ids = tokenizer(title, max_length=512, padding="longest").input_ids[0]
print(f"Tokens: {tokens}")
print(f"Token IDs: {token_ids}\n")
Using Pretrained Tokenizers
You can also directly load pretrained tokenizers:
from flash_tokenizer import BertTokenizerFlash
tokenizer = BertTokenizerFlash.from_pretrained('bert-base-multilingual-cased')
Comparing Accuracy
FlashTokenizer allows easy accuracy comparison with Hugging Face's tokenizer:
from flash_tokenizer import BertTokenizerFlash
from transformers import BertTokenizer
texts = ["Chess is Life.", "Dies Spiel ist ein Probierstein des Gehirns."]
flash_tokenizer = BertTokenizerFlash.from_pretrained('bert-base-multilingual-uncased', original=True)
hf_tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-uncased')
correct = 0
for text in texts:
flash_ids = flash_tokenizer(text, max_length=512).input_ids[0]
hf_ids = hf_tokenizer(text, max_length=512, return_tensors="np").input_ids[0].tolist()
correct += int(flash_ids == hf_ids)
accuracy = correct * 100 / len(texts)
print(f"Accuracy: {accuracy:.2f}%")
Support
FlashTokenizer is actively maintained and optimized. For issues, feature requests, or contributions, visit our GitHub repository.
Enjoy the fast and efficient tokenization with FlashTokenizer!
1 maintainer
Metadata
1 maintainer
Release files for flash-tokenizer 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| flash_tokenizer-1.2.0.tar.gz | 5.2 MB | Details |
Built distributions (wheels)
Total release size: 10.6 MB
Release files / flash_tokenizer-1.2.0.tar.gz
| Download URL | flash_tokenizer-1.2.0.tar.gz |
|---|---|
| Size | 5.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
079e4c0039ab56ccc7605511c816f1cfd04cbc074bc02ace775227398a19b7fa
|
|
BLAKE2b-256 checksum How to use checksums |
d80d2099e955e31901e425420ab20761150a69d07a3534edb10961817ad77a88
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp313-cp313-win_amd64.whl
| Download URL | flash_tokenizer-1.2.0-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 330.6 kB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
2b4489f0a3abc627af94bfb2df857c172760bd23c1a55bda297027132c855147
|
|
BLAKE2b-256 checksum How to use checksums |
360c0d66681223b52fc9cdda868ca486f0cf1d1d781b9efc852b783704ed46db
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp313-cp313-manylinux_2_28_x86_64.whl
| Download URL | flash_tokenizer-1.2.0-cp313-cp313-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 374.0 kB |
| Tags | CPython 3.13 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
a8ed41a3e88bac55c58deb109d690731bc95e603cc740c46e119ed1520d9f6a8
|
|
BLAKE2b-256 checksum How to use checksums |
7ec46f0166b4d3ed8110855d63912ff57db5a0ca5a3ceaa6628b2e00ba930966
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp313-cp313-macosx_15_0_arm64.whl
| Download URL | flash_tokenizer-1.2.0-cp313-cp313-macosx_15_0_arm64.whl |
|---|---|
| Size | 198.0 kB |
| Tags | CPython 3.13 macOS 15.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
6c0095cc4f09a37183b215e47fcd7d006e0600858b754272fd7c49b1887009b6
|
|
BLAKE2b-256 checksum How to use checksums |
924312437ceb4be15fe585c088793ee8b40cbc682496dda6b5bce3550cfaa5c9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp312-cp312-win_amd64.whl
| Download URL | flash_tokenizer-1.2.0-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 330.5 kB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
7c3c14ab3aecc912558fb4d1acf38c274dcf5cb82cd65889466e4fb4afebb90f
|
|
BLAKE2b-256 checksum How to use checksums |
5888316d17ec44786e9c7fd721905aaad7936cc8a4587400eff2a3070efcc7ad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp312-cp312-manylinux_2_28_x86_64.whl
| Download URL | flash_tokenizer-1.2.0-cp312-cp312-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 374.1 kB |
| Tags | CPython 3.12 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
ead3522450f97cb2d1291574a423ed231a22b056126713f1e7d425ae68118c51
|
|
BLAKE2b-256 checksum How to use checksums |
74633f71de5d16e204cee5bdceb39d875378c6a58f01435f758e2d461b89e274
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp312-cp312-macosx_15_0_arm64.whl
| Download URL | flash_tokenizer-1.2.0-cp312-cp312-macosx_15_0_arm64.whl |
|---|---|
| Size | 197.9 kB |
| Tags | CPython 3.12 macOS 15.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
0921dce5d3aac0187306e8bc3361f034a5db1999c3d5c42bd1e1314d4d344bcc
|
|
BLAKE2b-256 checksum How to use checksums |
500a0b6cc0cd6e4a9d6b456f962404d09c67c164b8bcf0c9711936914eb98902
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp311-cp311-win_amd64.whl
| Download URL | flash_tokenizer-1.2.0-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 329.9 kB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
3a1729827c3852eecec9bf993f0e77c6876c5899f439a32c6824b96795fedf4a
|
|
BLAKE2b-256 checksum How to use checksums |
915928a48b72af1b9fa97e594240ba5651fee6a0c562697e7121422582225b6a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp311-cp311-manylinux_2_28_x86_64.whl
| Download URL | flash_tokenizer-1.2.0-cp311-cp311-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 376.0 kB |
| Tags | CPython 3.11 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
326b97208df1c3516d40a1f71bbefd3c79b88a7b0a08cedf30c5e9c3e822032c
|
|
BLAKE2b-256 checksum How to use checksums |
04e33d3b6f5b1d4589c021189bc8c3f5ba2c6d34d879706bd619e2502174692b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp311-cp311-macosx_15_0_arm64.whl
| Download URL | flash_tokenizer-1.2.0-cp311-cp311-macosx_15_0_arm64.whl |
|---|---|
| Size | 198.4 kB |
| Tags | CPython 3.11 macOS 15.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
84281c8cb058d901fdde2f0d9248ef53b2fda889d7dcc7bdb3b4d7456aca3384
|
|
BLAKE2b-256 checksum How to use checksums |
59747745d51849643a1e76da08b8246debdd57e9ce9dd5f73cc00135cba66c2b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp310-cp310-win_amd64.whl
| Download URL | flash_tokenizer-1.2.0-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 328.4 kB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
5cac9365a26e62e325ecb93cb0433e8a4d9bcc1ad83f05c4b67185de50a4c64b
|
|
BLAKE2b-256 checksum How to use checksums |
e934ede484c78e4e0ccf0e5bc761d61961397e56e8853e6b5d792f5a40b4d3ee
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp310-cp310-manylinux_2_28_x86_64.whl
| Download URL | flash_tokenizer-1.2.0-cp310-cp310-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 371.0 kB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
961c57ca06dca8ffe30f214f772ca1fed1db8aca530d64351b759c29b2310dd8
|
|
BLAKE2b-256 checksum How to use checksums |
b6fa43ddc940587f302b8ceb058589b23b05ea62c334b94f865c0a6b5a3fe938
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp310-cp310-macosx_15_0_arm64.whl
| Download URL | flash_tokenizer-1.2.0-cp310-cp310-macosx_15_0_arm64.whl |
|---|---|
| Size | 197.0 kB |
| Tags | CPython 3.10 macOS 15.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
a2a48598f07e9f16935194f764f4f4fc68705c18d27c10e8b2d32c3007d08a20
|
|
BLAKE2b-256 checksum How to use checksums |
1ceab8cb319fc3ff9a6861580f3c761981b48d957d64abb1129fcffdbd70915c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp39-cp39-win_amd64.whl
| Download URL | flash_tokenizer-1.2.0-cp39-cp39-win_amd64.whl |
|---|---|
| Size | 327.9 kB |
| Tags | CPython 3.9 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
00ad2ff5c875f5fcd42b817097db9abd72c7115ca60eff20644048cb905ddae3
|
|
BLAKE2b-256 checksum How to use checksums |
7d98f081abdaf8134583c5c30aea621f6937c6a84035e501251190d9042fab06
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp39-cp39-manylinux_2_28_x86_64.whl
| Download URL | flash_tokenizer-1.2.0-cp39-cp39-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 370.1 kB |
| Tags | CPython 3.9 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
94c6b7ad6cf16998e5d9c5170527326c043d9afe1499cd2eec00bac98ae8a818
|
|
BLAKE2b-256 checksum How to use checksums |
0e0346caf34ffff2dd7f62ddbf396c2f9ad3403d45c0e1734cb3f77b5f994306
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp39-cp39-macosx_15_0_arm64.whl
| Download URL | flash_tokenizer-1.2.0-cp39-cp39-macosx_15_0_arm64.whl |
|---|---|
| Size | 197.1 kB |
| Tags | CPython 3.9 macOS 15.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
6fbf92bf992d7e65437dc3c531142e45f38c82995738e4fce2593de0f4745f62
|
|
BLAKE2b-256 checksum How to use checksums |
671061656b0394c42e7959fd3595ac8a7020b80f5831989914dc0a0bf02ef0b0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp38-cp38-win_amd64.whl
| Download URL | flash_tokenizer-1.2.0-cp38-cp38-win_amd64.whl |
|---|---|
| Size | 328.6 kB |
| Tags | CPython 3.8 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
f6ffcb0071950e1d25ae87f1574bb68993af1739012e095427f456732e143fef
|
|
BLAKE2b-256 checksum How to use checksums |
b0ecb7625164d8e8d388d47e7875b3723730b76584ed05839d9c7f7487b4e7f7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp38-cp38-manylinux_2_28_x86_64.whl
| Download URL | flash_tokenizer-1.2.0-cp38-cp38-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 370.4 kB |
| Tags | CPython 3.8 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
1191b44729004fbaae1e0220066120bbe0ed1efeb1a35287e94c27b48041f0f6
|
|
BLAKE2b-256 checksum How to use checksums |
241649699118f2561c1f75510ff081c900d1aab4fcde7a3b5ad086e51818e6e0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
Release files / flash_tokenizer-1.2.0-cp38-cp38-macosx_15_0_arm64.whl
| Download URL | flash_tokenizer-1.2.0-cp38-cp38-macosx_15_0_arm64.whl |
|---|---|
| Size | 196.8 kB |
| Tags | CPython 3.8 macOS 15.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
a867cb9b44a81d77e4fde3a8b1eef336cbfa0459dc6e81546825fa3692f3c6c3
|
|
BLAKE2b-256 checksum How to use checksums |
36a60f94b2ffd8a48fb37c366b829eda40df1983aec60f8fdd9f2a00e8ea1c22
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.2
|
