No project description provided
Project description
🪙 toktokenizer
toktokenizer is a BPE tokenizer implemented in rust and exposed in python using pyo3 bindings.
import toktokenizer as tok
bpe = tok.BPETokenizer.from_pretrained("wikibpe.json")
assert bpe.decode(bpe.encode("rust is pretty fun 🦀"))
Install toktokenizer from PyPI or from source
pip install toktokenizer
Performance
tok: 16.18MB/s tokenizers: 4.89MB/s tiktoken: 22.98MB/s
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
toktokenizer-0.1.0.tar.gz
(634.1 kB
view hashes)
Built Distributions
Close
Hashes for toktokenizer-0.1.0-cp312-cp312-macosx_10_12_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8ce080cd68fa1e18f16c0d1b9c4e49feb17026840962db07330705dbadc18855 |
|
| MD5 | f6a71cefb620ec6e5adb351b0356d1a7 |
|
| BLAKE2b-256 | f465d1f5a79064cd1545381df9db0c235f4102a50e5673d78d886df6cf12a2d3 |
Close
Hashes for toktokenizer-0.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a0f0677241a9b4a2ec966f55c2d8ffb434d9b5465a22bb885d6a41462420a381 |
|
| MD5 | 4f9bca01dcf7a3a621aaf5d54e3ba468 |
|
| BLAKE2b-256 | 6f2ebf88a95bc9b99c0c1b200d62659ece5431d10c7c19e1f77f45db798c3207 |
