Lightweight piece tokenization library
Project description
🥢 Curated Tokenizers
This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:
Tokenizer | Binding | Example model |
---|---|---|
BPE | sentencepiece | |
Byte BPE | Native | RoBERTa/GPT-2 |
Unigram | sentencepiece | XLM-RoBERTa |
Wordpiece | Native | BERT |
⚠️ Warning: experimental package
This package is experimental and it is likely that the APIs will change in incompatible ways.
⏳ Install
Curated tokenizers is availble through PyPI:
pip install curated_tokenizers
🚀 Quickstart
The best way to get started with curated tokenizers is through the
curated-transformers
library. curated-transformers
also provides functionality to load tokenization
models from Huggingface Hub.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Close
Hashes for curated_tokenizers-0.9.2-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc0072549c7598794355f219c4f12a0d09115eea64f4746ac0f46673e278939c |
|
MD5 | 5b996741bf808d52cee406229a14d32d |
|
BLAKE2b-256 | 878be4be39eae1f98d655c058776d780eb8f8dbc5ddd3db4935d74283d62606a |
Close
Hashes for curated_tokenizers-0.9.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 54efa11d957082a756cf21079baed7f0ebf043eed3a5f908c0a9615d19c598fa |
|
MD5 | de61aa5aacd99cd618c33ae60ca94d1d |
|
BLAKE2b-256 | 7e10cde9a6eabe8feecb180cc1d2e9bc5dea0c57a63517d656b98f32a870981d |
Close
Hashes for curated_tokenizers-0.9.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d775f76278f1be344cbbe2cba9fce9cb7567f7b224abf0c30a83f3be52b89eb7 |
|
MD5 | 2472572655dda9d0fdc701fd5403b193 |
|
BLAKE2b-256 | 9b64c9a74debf29ee9e3ef8afe0700d2fbf061bf22e5432caec0216ece1451bd |
Close
Hashes for curated_tokenizers-0.9.2-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 876f535ec3b3d45867b779d6f4ff909d9da73f2947bb1ac2dcfceb50674d7d19 |
|
MD5 | 8f9319484e1ab5b556cc7159e6b4a8bc |
|
BLAKE2b-256 | 42325f61a1c80882de3c856afe540f43ec0c47b262793e01b8cc81c68e0fefa3 |
Close
Hashes for curated_tokenizers-0.9.2-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fae145c600f00a610bce1d3bc2f835088ae14c437fec61ee8b501317f68e53b8 |
|
MD5 | 69d947da9867bfe0e467ce2c6ce21212 |
|
BLAKE2b-256 | 0d2f5c48d981fd9378e0abfe61bc6954d061ffdee623aa32be4122cffe44231b |
Close
Hashes for curated_tokenizers-0.9.2-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bab3d593929beac8d271b18d09d68d04924e49ea48fc0c9e830daeb6bf1ab545 |
|
MD5 | 5083bd02de5e7916088235d2f98f8492 |
|
BLAKE2b-256 | db97e5895591b16ece9f6cae36e5c90c9fb53ec077871fe4627d2d9ed1a38846 |
Close
Hashes for curated_tokenizers-0.9.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1a4b9e0bc883e224624471317b17d61521abb4c3bd6c690b42f6db8567e883a |
|
MD5 | addbf26346e23e29623439cf613ef41e |
|
BLAKE2b-256 | 1156be958002460b0f3aa798fef5f853c3e24007b7c6e57f4f0636a7daa38465 |
Close
Hashes for curated_tokenizers-0.9.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b832a564996420758f45c000fb56d578d9ea6c93c5abcaf18da8bfc2dc71fd53 |
|
MD5 | c96f815af07bcfd1f07177aac92277a0 |
|
BLAKE2b-256 | d25a7eccde6fea618a2b14932e8272c1bf9778f469166b5af5a7f5ced83152e6 |
Close
Hashes for curated_tokenizers-0.9.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41af771e44be868680d1bd7d4bc062fff5d890f7d81ba5c23097a451e7301c37 |
|
MD5 | f9c0578fb7348b1fc1676889f95b8a16 |
|
BLAKE2b-256 | 669f57bdd398931a8bc534c7ebc22a0c662da5ff562335a9d589b2cf553aa774 |
Close
Hashes for curated_tokenizers-0.9.2-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12bedbabbbdb3bcef5bad9e597d6f3fb74432d3bdd839a782ecb405bc5d1fad4 |
|
MD5 | 34d5fc110166963d521ae39b7d22b050 |
|
BLAKE2b-256 | e63558eb1367d2158db8d982dfc1602a3fbb2ebd7cba7d9fe49e59ec1cde37cc |
Close
Hashes for curated_tokenizers-0.9.2-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d9d655de6da771aad211aecdd787d229a300b93d9f43146419823d02b5b01430 |
|
MD5 | 776db5e0871764ade3ea514c9215882f |
|
BLAKE2b-256 | 19fdc293aae823f09e94b125b41d99525af9e21e947fb236504a8058b1727be6 |
Close
Hashes for curated_tokenizers-0.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 049387f0f433336b00806663496748a88f5589b977ffd00cb7fc1b70fea20659 |
|
MD5 | a3823e70408467ec125edc9c68a493ba |
|
BLAKE2b-256 | cffe4bed1fff3b0ef5b3e1bffcce924c0ae28fbec5de356c757e6517ebc0e29a |
Close
Hashes for curated_tokenizers-0.9.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d1a7dab4015c6bec7eb69953acddb651171e0c1cda553bae821785155e2c95b |
|
MD5 | 139f6502690dfb41d08290849123d64f |
|
BLAKE2b-256 | 321709c3866e126fd38d198b225ccf2cad857120a4e05b96111fa0763af8b425 |
Close
Hashes for curated_tokenizers-0.9.2-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aad34b1d63e9d20d95d535620e31c4fdb1269cf938a023813199e7197d284348 |
|
MD5 | 7b5ebdf559403e43911ef33cc5169aa2 |
|
BLAKE2b-256 | 4bd3ea6bdc52c458dd363539b0567ab0abed719662c7068891371f9f4aa862d7 |
Close
Hashes for curated_tokenizers-0.9.2-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 57fbd9380101750091f54bdc8e78eb084a0ce430338c3ee65ee8980d990c8c57 |
|
MD5 | b17b2d7c9b8ff6e543e93255a3423334 |
|
BLAKE2b-256 | 6f40470a09cfe6611cbfb7737722f5f8ed18ea4a5012000d799299a97e8a2eba |
Close
Hashes for curated_tokenizers-0.9.2-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3adeec8a4c4ac72fd8f9750392b96ad58995cc1f93cabc583c9053d3c6aa6631 |
|
MD5 | eb7cc872b7e5f374c90016cd2e64e7c5 |
|
BLAKE2b-256 | f7e693ca07697f4bf1868b614bfb601405d79dc64cc336a2f52aafa4b75c2f6d |
Close
Hashes for curated_tokenizers-0.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 55538361007a0a11e7bf83023c2e1950086d698d327691610771367dfbb281ff |
|
MD5 | ed2911c9db913933bfbe8d5aefdc926e |
|
BLAKE2b-256 | d819110fae679e14dbcfddaa4d02fd892602067a3d43e8e9b2ccfa41434096d1 |
Close
Hashes for curated_tokenizers-0.9.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 71ba80d589797e8f988e31e80d6d2da06bab54af6efdce42d01907fe98759f58 |
|
MD5 | 17683be8bb61f0c47adfbf9982ca60da |
|
BLAKE2b-256 | bfef43d853ad64e832e55b70ec7f42a732bf7e47f158c2f2c532abe287d535a8 |
Close
Hashes for curated_tokenizers-0.9.2-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a0731e84262bdd028551de9cb4ad2c6820885d418a0874d8496d1a7518761312 |
|
MD5 | 406d6b807bba558d4c9da166eb03534c |
|
BLAKE2b-256 | 9f19292514662180e948f967c26e9e53c9cdcb52a8bf419105e936790828362c |
Close
Hashes for curated_tokenizers-0.9.2-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 61bd5c652eb8521ade92e078bba11ec2b2d6784e448820d5ad6ee95ef7047a7a |
|
MD5 | bd7264108b5c94060767b099b09a9758 |
|
BLAKE2b-256 | 3811cff1585f2f34bf029bd2748f45dfb04ed0711287380f2b44f2eabf6e26d6 |
Close
Hashes for curated_tokenizers-0.9.2-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1aaf83005d1f44825a09985b0abed3b11d7869bf98fe658bdd8893ba7b09350c |
|
MD5 | 8dec2d03f60cabf2aa372b386244179a |
|
BLAKE2b-256 | 45b7b0ba95b73a8152a9dce9c22e8332f882482e0c237c83194b8dc6d1688dec |
Close
Hashes for curated_tokenizers-0.9.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7af0caa40287954c93357ae5c767928786a8b4a22ab87d74d5273778d36bbc74 |
|
MD5 | c4a9554ae985a2cf87b0be06d8e82bf4 |
|
BLAKE2b-256 | 78ab846c3f24e5386dd5bce6f73c6c79f009ff2fdf9522c42cf59821cd68966e |
Close
Hashes for curated_tokenizers-0.9.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d639e08b28cc085f1da933f2098d38ac16a81c81c16a081124110975b78be94a |
|
MD5 | b9fdf1192a841c6b58a4d7f1bf1b2e66 |
|
BLAKE2b-256 | f71d22918e984b03a51ac54882f182ae5db83e73082c5ae85f146d07c935856e |
Close
Hashes for curated_tokenizers-0.9.2-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 473f06c0b332854c5cb0ad53d4dc41af784167287d25ff0952272a60f33e4fd2 |
|
MD5 | 9b6470463deb1894103af6f1427e0e64 |
|
BLAKE2b-256 | 09af99c2694e9a7f6bb266fd2e90601d45e5807d25372136bd743bc95126cb45 |
Close
Hashes for curated_tokenizers-0.9.2-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c4e453ff10fc58b9d2658b9bc34aa4b524f87d78d07dfb76c533c3d43485c5a7 |
|
MD5 | f22ce609b7c19637bba37c189f343dfa |
|
BLAKE2b-256 | bb62730f9b252e3f2572bb14ef69758cc73ac1a1608638d1b7268798693ea9e6 |