Lightweight piece tokenization library
Project description
🥢 Curated Tokenizers
This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:
Tokenizer | Binding | Example model |
---|---|---|
BPE | sentencepiece | |
Byte BPE | Native | RoBERTa/GPT-2 |
Unigram | sentencepiece | XLM-RoBERTa |
Wordpiece | Native | BERT |
⚠️ Warning: experimental package
This package is experimental and it is likely that the APIs will change in incompatible ways.
⏳ Install
Curated tokenizers is availble through PyPI:
pip install curated_tokenizers
🚀 Quickstart
The best way to get started with curated tokenizers is through the
curated-transformers
library. curated-transformers
also provides functionality to load tokenization
models from Huggingface Hub.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Close
Hashes for curated_tokenizers-0.0.9-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 209d756694c7fb000a0b642016eb6e71c740cfce293adcbf3384aa2a1e701eb2 |
|
MD5 | 248180531c8d798626727312f7e09ed8 |
|
BLAKE2b-256 | 0ffc776b7464029ea126bf55df2a5edd437178ad8e5c0126f953891dfa603f9c |
Close
Hashes for curated_tokenizers-0.0.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 35fb208a01f2b3f22172596915d229859549a2d76e484be976dd728b1ca3bdec |
|
MD5 | 62b772af4212d2a4e73666e7fd10c58c |
|
BLAKE2b-256 | 212412ae8f92d0e319ed07dd9c3ee5d24e71dd6ff3dd8d4dbe2126a6e5cbf7a1 |
Close
Hashes for curated_tokenizers-0.0.9-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 64b9991a9720a0ce8cc72d29791fd73f2cc2bef0241b002fd2a756ec8a629143 |
|
MD5 | 7d15949338d1a897fd3a7eebdcb0f4df |
|
BLAKE2b-256 | 1b7c2d24633275f2854c144652ee6ef97ae85d444855b6da5aa1203678541fa5 |
Close
Hashes for curated_tokenizers-0.0.9-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2abbb571666a9c9b3a15f9df022e25ed1137e9fa8346788aaa747c00f940a3c6 |
|
MD5 | 8b19d39834dfb79b94bd85bdb5dcbe25 |
|
BLAKE2b-256 | 34fbd6e57b1155bee398f43de58ecdcdda44957e9635183312ac0820a19fc94d |
Close
Hashes for curated_tokenizers-0.0.9-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e66aedfeae0c91f3f3e2980b17933b3d08f3fba6c8ba7057b9b05d596e8a0b27 |
|
MD5 | 78c865611f39a248b313b61805feda3f |
|
BLAKE2b-256 | 103ec10474a21ed0166f94cebb46fe96cf07fdf7f399d84e6157ec4dfbd97b53 |
Close
Hashes for curated_tokenizers-0.0.9-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c661d09ffe7a4a9175f28d76034e01c87df9df6fedb998151abbf201f28f1aa0 |
|
MD5 | c3b438c9437f79d789c83b1bd22ba2ed |
|
BLAKE2b-256 | 75efea0e193b1775688263ac9261128b616cbc11cb052feb068b4974626d2715 |
Close
Hashes for curated_tokenizers-0.0.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4079b1cb2220cb76deb271fa55f4867be3764f15e8fdb1bfc0a2041081570224 |
|
MD5 | 1e4294a7f3a6559063ae3bf244ce9282 |
|
BLAKE2b-256 | 316970f6295dd03fed67afa8520b066026764d719fe123ddd108137ee4c9a732 |
Close
Hashes for curated_tokenizers-0.0.9-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76ff13e8c19f7cdb03441ca5ec9ce85f133da7fd5b9cc574d8d18af41ba8a50a |
|
MD5 | d361ba9a4a08ab6a4b143fbafdde5f16 |
|
BLAKE2b-256 | a9c752452cf5f200f95711c15b474d6230fed40330621c0e423c4ce7e02af1fd |
Close
Hashes for curated_tokenizers-0.0.9-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d1fef3e861df50bd6337364a87f447fbd0a6f01c095cec121b7404d15512138 |
|
MD5 | 95bb2b78c6e0dab857f828e5436c5c2d |
|
BLAKE2b-256 | 38e388c6681df8319fef9670c99e8dafbc3e89403f199cf6d009a407856e9ebc |
Close
Hashes for curated_tokenizers-0.0.9-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 899128d78177ca0ac668addc33b430020f737dd08bc6bf3753ff4d9ba0e41e75 |
|
MD5 | d4936b1a7974482e6c4b43773c414582 |
|
BLAKE2b-256 | 1ff2e4fa27a078ca6d7db87f82124695ce8822104285d4f8b3ec9900ab18c2df |
Close
Hashes for curated_tokenizers-0.0.9-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00a9eff167481494f967ad0efc5c53164d460d4f40d816f6c132f69c8584a735 |
|
MD5 | 40b7905275636a1993a043ed3f652c64 |
|
BLAKE2b-256 | e14e108a03b27d7e3646a9f74c73efbf8d94feda16e22d49b35d198814f3e13a |
Close
Hashes for curated_tokenizers-0.0.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a61acd1c66aea2198702b2a1418a6f3bf1241e3e302c1295a5878e892010642 |
|
MD5 | 9940f9fef7c663e696335c314f83da23 |
|
BLAKE2b-256 | bcb3c1ef2c0587a926a2a4f2fec4ea8338e34068845decbfc64e5f554b5d01a0 |
Close
Hashes for curated_tokenizers-0.0.9-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2cfc4541c3e5738d74dbf859eb87c26112178b7a91be1d99a4bdced8182f4a73 |
|
MD5 | b287762507122cc0f0081b92620d2240 |
|
BLAKE2b-256 | 54f954b7f83a6fbb3d34e45aa1a095c743b174186b28d375714b87b48accaf89 |
Close
Hashes for curated_tokenizers-0.0.9-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 799b8a9a1603b7d12683017409bf338bff925aa9806fbad0925ac550501afdf8 |
|
MD5 | b6365a3bb1e770114212c151b681b123 |
|
BLAKE2b-256 | ae9b3862be9b9bc97bedfd159fc30ff81f531132de59e324b9b41c264702cbf7 |
Close
Hashes for curated_tokenizers-0.0.9-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19d3a2570dbbd08bbdae4c79d187fb150ea3b663c2f060bd1e4a050a1358cfd1 |
|
MD5 | 82e176d1a6231b20b081b085f61d1ed4 |
|
BLAKE2b-256 | 016407c176505994cdd3ea3d7b1e56ccaa0f14f506be72dc5bad9a627995f048 |
Close
Hashes for curated_tokenizers-0.0.9-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 81e90b046cb439466e7f3775619f87036a94c1d851146b45e5c63277b2d9ee07 |
|
MD5 | 3f020be337d9044d6f74f964603d39f0 |
|
BLAKE2b-256 | e9ba394c808069db062412dce701a9b3242b97589073348a1ec44e26de2901d5 |
Close
Hashes for curated_tokenizers-0.0.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4307cab745488937b66948a1ce4bf86ad978c9fed29c93f2a72882b32d344689 |
|
MD5 | ad52fdaaa5dfb3c8b0ff837bc192def0 |
|
BLAKE2b-256 | 923ed1c930a002b640af74c37fc4cc4bd64dd9f8842a07876aaf1db23d6d1185 |
Close
Hashes for curated_tokenizers-0.0.9-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47a1219418654a26c1e039d08f37246d9917af4c0bf492441095f4a5c1072ee2 |
|
MD5 | a5b5887729e39009dff9d807f723566e |
|
BLAKE2b-256 | ed897bb5dfd391911127874c3655cbc1704b23f3f5da2721e8b3e7441d382535 |
Close
Hashes for curated_tokenizers-0.0.9-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce0b2369783a4645a3806d94ec42c523de4e2e23932200f452e05d3b2ea81481 |
|
MD5 | e4b1401ff25ad8ba6aea36004e71dbaa |
|
BLAKE2b-256 | c86a9c090065f43f492420bc04684c7ca6700aaacc156d25f882fa09f42de51a |
Close
Hashes for curated_tokenizers-0.0.9-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fafa009ee45d2599bf321c194a6a71c05372fb2d325640458d0d70320229b8ff |
|
MD5 | 326dc91e6aff735e11330bab439a174e |
|
BLAKE2b-256 | 001f84c8ce42b4e8c2f2105eaccf52f21050de7419212af372ca9a38a2c8b0d5 |
Close
Hashes for curated_tokenizers-0.0.9-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6aa7972ebcab5fa3486b4b0f991bd0d3ba7b7d2643508ac8bdaf8af5701d7ce |
|
MD5 | 420396a52408a31e0e3026ccb62d84ed |
|
BLAKE2b-256 | e22650069930bbeec8616fab0b22d5072e6924f37b4ec01a171c16bc7c92c976 |
Close
Hashes for curated_tokenizers-0.0.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41b830f3f6d493adfd0ef9fe793ee07ca2a47d6198f95b3dabe3bc9600af3cc4 |
|
MD5 | 034b8e90db18519fafd8e9aa89fc40ad |
|
BLAKE2b-256 | 2a9ab56b81ccae5bccf5060b8ebbdb06b94a04e24a88da34cb3b55742103fea6 |
Close
Hashes for curated_tokenizers-0.0.9-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ffd667d08bf7c560afd02998aab8fcb5aaa16c9c69cf79d9e4ac684196a5b046 |
|
MD5 | ea569369c5f0b89ec3b48e4425903d50 |
|
BLAKE2b-256 | b1b91f780c1a0a7b30ba5ef9bf9e89536b55c286110ef580e17352d211bdfc91 |
Close
Hashes for curated_tokenizers-0.0.9-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a5df6be9f1d1de9659b1fa34f2b53449c405097816cab5eb505a02d4767dd9e |
|
MD5 | 51517b2709317a98965fa4fcb0459ad6 |
|
BLAKE2b-256 | 03324f3110c6a75998a6b451ad56adcc0b747f11a0abb0877dc87b137373b1f8 |
Close
Hashes for curated_tokenizers-0.0.9-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 28613e6a78e3456f5683e8f05f180a5aab0f18732282de8a236c87d66ff6e9d1 |
|
MD5 | df9f5d7fc92ec5c0540721e945020ef0 |
|
BLAKE2b-256 | 00720587b2345e1b43c4872b541f400db57f5b2df2831b7ffb6bd0e93dd44c59 |
Close
Hashes for curated_tokenizers-0.0.9-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ed857074ea7362755ffaa027ecbcc047f78dc461bba3ba9e1dfa00f433b7e4b0 |
|
MD5 | 172cd321bb110b213322a92411567c78 |
|
BLAKE2b-256 | 41708ad75360ed76f3fe97a7ac25b0435bb4a3a20c5820599c72d357a00a75a9 |
Close
Hashes for curated_tokenizers-0.0.9-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d135d80ed31a7b10b192f8a26147e4b95854f11963f2f6fdd4c95d870c36fd11 |
|
MD5 | dfc0d28c39547913742e3b7d1db7324a |
|
BLAKE2b-256 | e359b64ec3081e6ff85d1bc6c354a7ced8b86dcfeac449b70d22a85f523e7d89 |
Close
Hashes for curated_tokenizers-0.0.9-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ad60ace40c52c2eb447b0ca0cb1234a57a704641eb95344bbaf6239d359dae9 |
|
MD5 | 58cdfd27590846e77d4afc91fae25830 |
|
BLAKE2b-256 | a452dc60d8e15a92088783aeb387de9bcef304a81656a5ab8c7ded4c5411ba94 |
Close
Hashes for curated_tokenizers-0.0.9-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a94dcc4ef63bfd7e05bf11f3081fd472e241e70230207a6c8087fa778fff6a9 |
|
MD5 | ec3d496c33dd03c45980988654feaebd |
|
BLAKE2b-256 | 7d27af81a6e94fd546c98cbcca17dcc7e6f99d283cb0692209c5afe65adb36ca |
Close
Hashes for curated_tokenizers-0.0.9-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df353261c37c5cc70c9e2a64adb88bcad847c171aa40ed2734fce70796fc2e1e |
|
MD5 | 49c141dc9cfd345a7252071a82a03a95 |
|
BLAKE2b-256 | 19dc0a44e8c619a85b5de3abc881c1bdfb6a148dd277dfb583eeacdfe6c9f4e2 |
Close
Hashes for curated_tokenizers-0.0.9-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 009fc6f30b2e38a7d2b8ad8be47816b78c53225c0985b97ecffc45806311b0f2 |
|
MD5 | 91baafa0d8c86347462dfddb3cb7512d |
|
BLAKE2b-256 | e7343f3aca94afa7854030c96cfcff07e9898204baf090e3eaae0fa1ceb12e69 |
Close
Hashes for curated_tokenizers-0.0.9-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66c29358530c3fa7a3e25cd05c0d6d5fefa4807e02265e0803503f9cc98ab8fc |
|
MD5 | f3b80944926e2dc498c0bc0cbcafdc93 |
|
BLAKE2b-256 | 16345bb6220034b24d6601a0c3169f0b0f558522a48c2bf8b591b6bae11fb3a6 |
Close
Hashes for curated_tokenizers-0.0.9-cp36-cp36m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04c44758348515dcd36b63cda6b3417ce028d150695d4d2cc16022fa5fab41dd |
|
MD5 | 0e235eb19789010d3548be3123b49187 |
|
BLAKE2b-256 | acd3b0054e3ebb6937a84da8a3822fa926fbe18ce6eb07cd003e170d4dc00bd1 |