Lightweight piece tokenization library
Project description
🥢 Curated Tokenizers
This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:
Tokenizer | Binding | Example model |
---|---|---|
BPE | sentencepiece | |
Byte BPE | Native | RoBERTa/GPT-2 |
Unigram | sentencepiece | XLM-RoBERTa |
Wordpiece | Native | BERT |
⚠️ Warning: experimental package
This package is experimental and it is likely that the APIs will change in incompatible ways.
⏳ Install
Curated tokenizers is availble through PyPI:
pip install curated_tokenizers
🚀 Quickstart
The best way to get started with curated tokenizers is through the
curated-transformers
library. curated-transformers
also provides functionality to load tokenization
models from Huggingface Hub.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Close
Hashes for curated-tokenizers-2.0.0.dev0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 73c288990b3e288015c36aa058b6193a734fd9034e20cb93069e6472f8d67f49 |
|
MD5 | 2d50af31ca1469335e74e44b46c8c2ee |
|
BLAKE2b-256 | 9035cd9bf566f3991f993f04b506355b0fdbeb0cebd50201a3a5274565c9a342 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 654163266f4ace4a57288e847e10a03ce72c852b0fb3a54d832655db0d4dc599 |
|
MD5 | c026088b987cf9815b10b59d473aac27 |
|
BLAKE2b-256 | efadcd3abad4896eda57fa17b963b7a543da1bd127a58f4677d29dd9cd5ac827 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c61e122a4373388869a00b6e3a6407df74280f3f9dfe1d64fa8fd9a1c4bb878f |
|
MD5 | ea75d8b5ff924d329e490b5d7ab1ac39 |
|
BLAKE2b-256 | c52acbe8bd31a3dfc5207a7547262ac6ff3fd22e9cd1de0408c20d5652a188cb |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 16bb5dddb56d2de7455aae24d1bdda7ab35d7dcd1282dd42a5bc9547ae27d45f |
|
MD5 | 88662e6c267221b1fddd6f49295597a0 |
|
BLAKE2b-256 | a50d59e67ddd3db7a324866b8690249f043e3cf5f1c030785ab3ec3d4f13df53 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f534d309a3f1478614e4b6ba3aa8fe0627203fb242c0f283e75413d120819ff1 |
|
MD5 | b3bda7724f5467ef11bbd7f70a081b07 |
|
BLAKE2b-256 | 2de153cfbd8d23cecd457cc15e2f8c2af90bd1f42e4617fdf93b425cc06f3a40 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2f2ea37f36f14f2342838baf0bc9b822897fc98f9b160941abe24cc411477ef |
|
MD5 | 8f0155212fd3b39a28a245af56ae20ac |
|
BLAKE2b-256 | 38b402786e53fe14ae2e2285437cfc8dcc8b0ef768193880d01bbd33844cf31c |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d6e5b0c6e55e2cd30cb8b936f406352cc12b1454c0eeb6cf53370d1e7aed51b8 |
|
MD5 | a3072a3e13b70c9e1eaf73d7e63200f0 |
|
BLAKE2b-256 | 41ec384464cce9006050a2e1894d6f05900e99018f6f07f3e39a8b28f3067d9c |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d6c3a858fe52085b8e188ecd6e5ee2fe6d45fa4ff71745efade422b73e1a284 |
|
MD5 | 7c723c96bb141f1a4bf04e00434c7a8a |
|
BLAKE2b-256 | 257dcc79ee1d0ccf5d2450ce5b35186af2fe303228af36b1bfcc3a75abc0a58d |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6f4d09a8daf63e3c8e6724795a7f78c243dc781bed545a48d27f0b572f72119 |
|
MD5 | 1cdba9de246cab1fbbd1028530106087 |
|
BLAKE2b-256 | 8dd102b653995d90899b88a8d8ba6d79a4e3a53bb5331fdd5d0c541e3ea8332c |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dda17f87265b0f0d6aff460c1c54cb2de640409035abb80b0781b4d80df42adb |
|
MD5 | 49b891b354a82bafe3078dae843f3c92 |
|
BLAKE2b-256 | 3ee5b591157291a1a034b9b64609fe8b72d2f36d5742f54bdf6e602213a5d156 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a51fa35b2ccb22e50088fc9ca64f5cb9de6fe947e5b04f125f120189f9b6518 |
|
MD5 | 939711619a73e078e8697e1e894c5ac9 |
|
BLAKE2b-256 | 1f0b1b5b0682057c6cc8b4f3a2c8ab9df906faed19a9274c84bf73f319a59151 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c78f2b3d8dc67651cbac58963f18a15435ad06db1fabf54ed0c9cdde94552a1 |
|
MD5 | 4527d9b3b64e3e77c0d62e2eb7049ea0 |
|
BLAKE2b-256 | 1b0329903cc1bc43f2e9fe5b505e7e679f9435374d313d6db80bfb46f65db0ca |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be1e79a89f8bbde8aa99ff43921001f944551eecee6c86d1c8c804463638fc05 |
|
MD5 | 01f285799f300afa6e0a06ba949c5147 |
|
BLAKE2b-256 | 9b3d8c3addf17112bdc65377b411b49f35ae588dc0317f81fb9c31ef65027f49 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d37f6540166c18c381cd7a285e6edcac36d342f38d8c8c22b903f81e25e13e3 |
|
MD5 | 9d7ef18c840054c6a9ff4cc05a690117 |
|
BLAKE2b-256 | 7b233d3c80ce79db309651e9ad12cd8b0dfd0428ccebfb73b32d90c131ec3202 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 242592c92782bf61fb8785bca1551fd02658ef112ea1619ea8f014cf4cd657ee |
|
MD5 | ea2a89f85136c228b3044aa9a326b1b0 |
|
BLAKE2b-256 | efb1a20c1eb69433f0ed18351b1474500eccde0ea69ec92fe17146772fe315da |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7dcc0869dbc589282f3d5958531734e7d6f88f586e6a48a01b5406c4574d9cf4 |
|
MD5 | 1b65d7090c41d6966964773034423996 |
|
BLAKE2b-256 | ff04a30792d0b01756d2479006483610d62ef085713faeb3048fb677b949f956 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9a2646358b9eb5c05316582007cbf2cffd0f3b70e7488172414b96e26b02104 |
|
MD5 | fda2835b906b245ed38e8c8aa231c00e |
|
BLAKE2b-256 | 64998f821850140e22c1b22d7e075ae45d3c15a03a59c2359572dbb56134f6a5 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a0c825194bf89f88cb15946e85cdc597b9a2a32267f3fd20e9872a2d36d08553 |
|
MD5 | a972f11e8056462b66d34f3de317df67 |
|
BLAKE2b-256 | d20600591cf09dd257784c6c6b7c536c7dc3d179556570f22976846a65dacc61 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3bf9a597f93222c4e3d38b9cb73c0ee1ad1e70225e5d66ae455914073f2f0d30 |
|
MD5 | 5793d5ca5cac64faa7d2f9d9779968f1 |
|
BLAKE2b-256 | ab4ed508c6724f8d963f8b602eb5bd39486a273dcb8db139f1070e2056898c57 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6aea51d28c7e7f6877301d91e25b912c6a3acbe21c95ade7c2a965f619566e94 |
|
MD5 | 71b58aedffa6c8fff4f1bf1fa6b70286 |
|
BLAKE2b-256 | af2af47fdb8c4a3d070b469735b2c275081861eca5d5e6bc24982e9a89ebb305 |
Close
Hashes for curated_tokenizers-2.0.0.dev0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | edd251a3ce5b24e8d22a98cd0305df6e63649ef331186265d94b2fe5c50e0500 |
|
MD5 | cf971c713a645ad25a8b44f14752e26e |
|
BLAKE2b-256 | bfe5e186e44190c70408bd73cf81ea795d8dd5ce4494e29fc73e6b58f2e17af0 |