Lightweight piece tokenization library
Project description
🥢 Curated Tokenizers
This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:
Tokenizer | Binding | Example model |
---|---|---|
BPE | sentencepiece | |
Byte BPE | Native | RoBERTa/GPT-2 |
Unigram | sentencepiece | XLM-RoBERTa |
Wordpiece | Native | BERT |
⚠️ Warning: experimental package
This package is experimental and it is likely that the APIs will change in incompatible ways.
⏳ Install
Curated tokenizers is availble through PyPI:
pip install curated_tokenizers
🚀 Quickstart
The best way to get started with curated tokenizers is through the
curated-transformers
library. curated-transformers
also provides functionality to load tokenization
models from Huggingface Hub.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Close
Hashes for curated_tokenizers-0.9.1-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e00968cfe99273f02885e9dd6a060107416e8339522a248a84a7abcfe1ad5036 |
|
MD5 | a331e5e6b4ca865aa5ddc8350802d0f3 |
|
BLAKE2b-256 | b9a9e05c583e4c322b341d5cf617fd69f5a9c862831ff98c1b8629ee610cef87 |
Close
Hashes for curated_tokenizers-0.9.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25f4e0a54d91992c4fa337acc60a8612ad8207bc44e1d3ed1418464975c44a0b |
|
MD5 | 6cb4fbde2f0216641f2da8f405022971 |
|
BLAKE2b-256 | 9a5533660f32b702d4aea6ad119ed69e646e9409018a1592243e2f6842ad1254 |
Close
Hashes for curated_tokenizers-0.9.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a4380b699dc933d9e9b399e755c76e4fac92f4a791cd1cdf0a150ba0b9750d9 |
|
MD5 | 288ad4ceb27525ba0146bd50ba173f2b |
|
BLAKE2b-256 | be125fcca75a05be87c75a32dfb49e6169a1b91a8f9736152e9c4ee8f6709054 |
Close
Hashes for curated_tokenizers-0.9.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7241e4094e02a84520b3fda1f06cae1d57ab0d70160e3ac9b3fa6794854f6e96 |
|
MD5 | e66a0d4f90a616d2ff15b6b884f9135f |
|
BLAKE2b-256 | 944de7d4ce841c41839a10a26ad7cffb4c5684de3ba457f035b3f47d45042a12 |
Close
Hashes for curated_tokenizers-0.9.1-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 63e05eb03e5e3791402379d03c845779be3bee08ff3d5eb888ef48eddee453fc |
|
MD5 | 0540077048e1e9b96f18d846e830ef3e |
|
BLAKE2b-256 | 9a2da8c00d089e88c1ba386b50869631f649e048e726bffa7d856a24919c093c |
Close
Hashes for curated_tokenizers-0.9.1-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 793e0d82dbc2de47a43826d44fb143c0bdfad6823026bcc755221aed553b0a5d |
|
MD5 | 4358914098ef3c11bac58c0c3f22c7df |
|
BLAKE2b-256 | cb94208e45094651989d5be9f8e13a78d2faea3dc2c21781571b511ebaf7227e |
Close
Hashes for curated_tokenizers-0.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 57079c6c3485fed9e7da233a8570b807be0e8e42bd95e44631fe417f0e8064be |
|
MD5 | 3829af759decc7029a74eefb4caaa349 |
|
BLAKE2b-256 | 26d6e898436ca5655d9e37ca4b9fc8fc92af46c41cef3c39ae8ab2f065f1867f |
Close
Hashes for curated_tokenizers-0.9.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 243aa6e74aaf2fc24112cbbea87180620ffde14174f2f09b71823f2c0deefada |
|
MD5 | e1f9c78f6aa9cd1ba40e30bac5332af7 |
|
BLAKE2b-256 | 2415227d9be7d2507ff56db82e37fb5ce72c5a522d05d58ee53768cbd8ad38f1 |
Close
Hashes for curated_tokenizers-0.9.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 73bb4e7a4f087979fb2959ab97245f1e3785b940eb4af8c403cd6ceecdcb8617 |
|
MD5 | 0ad7d735a1148a56092ce2d965b1e70e |
|
BLAKE2b-256 | 1d9294153177cb3d67f102b1ae6f8f471ce61910da110f541aa21c590d0fa9ea |
Close
Hashes for curated_tokenizers-0.9.1-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66aaf450ba35b240bb7cf671f32fe5246f84d2cc7e6cf308f9cc845c5bbed2a1 |
|
MD5 | b4d179847c08e2709c247f62508a5e2c |
|
BLAKE2b-256 | 066ac304b4d9c788ca12f677052e0750a8d03827c8d9ca8be33e7eb6ed3e76e7 |
Close
Hashes for curated_tokenizers-0.9.1-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c001db4c3def3cf34fdd3421471b83692f90a2aa0ba9c157db80b77c90aea6e4 |
|
MD5 | 9a4a23710003fca1edca5887910208ea |
|
BLAKE2b-256 | d8c62d3a02bfff7c55d7eb618a2b37430edb9cf6888d3aaaf9fd161124a33f3b |
Close
Hashes for curated_tokenizers-0.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a0772b8688dd3a3aee857fe490b067ee2776617536a7892c4729302f04dd786 |
|
MD5 | 34eba741d68863b21e1e8edc57318eb5 |
|
BLAKE2b-256 | 4b5ca05db29de7f4164405a73a670d7f50b7c3bf37c38f688ac80e37edaea97b |
Close
Hashes for curated_tokenizers-0.9.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25b198fed8a79404024b338e5bb0d40be259fc81848a75267280fbe065c64204 |
|
MD5 | c882c55c50d6b98ec7964ebb6beef05d |
|
BLAKE2b-256 | 974fc31802836681c62d1cd8120f5a48206db6a11114cd55a61897543cb3c6b2 |
Close
Hashes for curated_tokenizers-0.9.1-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd7d91728c55e22343a69c734efc8552f795f7c57e784eac43a45f07c41b6c5f |
|
MD5 | 5f004e322921d16bd36a25d6cd569ae5 |
|
BLAKE2b-256 | 1ef15912afa955de2f2470e9a78917d50cd7df76592625bb145b988c5f69cb90 |
Close
Hashes for curated_tokenizers-0.9.1-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b35046d47f918246bde2ce0d938c267ee41a652f80b28ee9b449810ea70eee7a |
|
MD5 | 884a60a7aa5ad57d3c8fbe60dd7f103b |
|
BLAKE2b-256 | 675f3d29e610d7380ba135a8af908201d3dfba9c0f0d3f497f23a01db176fa47 |
Close
Hashes for curated_tokenizers-0.9.1-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 940948eff3534ace43e85d3cbc3b344c411f688596dbcc8ede0c145b760ce784 |
|
MD5 | 32f9419836991bd17f6a4a17a7e6c670 |
|
BLAKE2b-256 | 24c80aa3ff6a94650b146064f57f97449c83c3d7ef941f89492e50a5b4ca9b39 |
Close
Hashes for curated_tokenizers-0.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8427fb9143ca29c2eb1626c8698fd9c3a64c3d5b4018c6a17ce91571443da322 |
|
MD5 | 069a2a5a31604c7a5edd4c3ecf8d1f2f |
|
BLAKE2b-256 | 6df1b0ce1ede783bf722e3556bf2913819647f3ecd1b81b73447f9bdab0c8464 |
Close
Hashes for curated_tokenizers-0.9.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9a6ec75ae075d43d16adc9e03ab31e186f0cdfd42fcf728d554281991666f50 |
|
MD5 | e2cc5859253e654a54240f7f09cbc2c2 |
|
BLAKE2b-256 | 08ee55d78ac3727bdc48927eb2d17dbd78e34e274dcdd0a59807233f7d2753a4 |
Close
Hashes for curated_tokenizers-0.9.1-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 05381ee89726a07005e70e3c873569f30d2ddafd822620f0d1fde9916ad0ad24 |
|
MD5 | adba55a8e7890e28dc3eb98a60a8ff0f |
|
BLAKE2b-256 | ad43e0013fdc812f566a5ad8cecef916da8ae107c28f401cc6d42c03c61ca560 |
Close
Hashes for curated_tokenizers-0.9.1-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 302dbad5f78c20d52a82ea9674d72369c76ec73f68fa0f90fba7e1f807134ff7 |
|
MD5 | 8d04f150b24b678709bbfd9edb203caf |
|
BLAKE2b-256 | 4a0619df83a103277e5adb477a4ef75208cbe092af2d1b66eb55f5417a6b7ea6 |
Close
Hashes for curated_tokenizers-0.9.1-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 044e2271f7f23aa4fed6b875b3d111e78bda99409d90823c1a4ba8a5f50af57d |
|
MD5 | a9bf9da01f44e805ca7f9c3f3164dba2 |
|
BLAKE2b-256 | 32986228081c15b57b047b825ea072a96c906c18307ee84ec404cd15fe6afd40 |
Close
Hashes for curated_tokenizers-0.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 02c77124b8379e6e7ca02e9026269e037874e853bc4b4abdfef84214527e3502 |
|
MD5 | db8111451083d060ad59e36393c7b945 |
|
BLAKE2b-256 | 306d114eb2f4b694dc163872980c323a981528d5554cc36c720b4ba81902d9ff |
Close
Hashes for curated_tokenizers-0.9.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 56d7b146386c8627c0f6774485492621e0a20e4a1c00eac18a3aca801afc8e9a |
|
MD5 | 52d8fc98b602878d84886f77a6eb9a2f |
|
BLAKE2b-256 | 4a747151b1b036b1e873c095357988f4d7327010a476396673eb74496c8ba62e |
Close
Hashes for curated_tokenizers-0.9.1-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7338c66e786ab51d264a0c574b925c05fde65fc1670958e70b359aabf50fb29a |
|
MD5 | d4f0e1fabf4458266a205f570149e4c0 |
|
BLAKE2b-256 | 0ce526693e25450a5135390e5aa5b06b1dc635d6179b4d0a428f54a1dda807ef |
Close
Hashes for curated_tokenizers-0.9.1-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 21820589d9492f53dd9ef42615358e841b4d79d8bca8de9dad1f4f12ad676861 |
|
MD5 | bdaa4607660afa0abcf45aee478cabce |
|
BLAKE2b-256 | 8edf35f79ceca35be049f0e3b8c9bd38ad28091fae8188e35d288e07e35cf325 |
Close
Hashes for curated_tokenizers-0.9.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0df70987e240a9d7ed177e3057c1f4041ae232bf7d40340bb40b74371560cc82 |
|
MD5 | a284e31766cb9855cbc7fd4239c3644a |
|
BLAKE2b-256 | 0f4a57a12a720a012f63550ceb63d7aadcb4357eb9c7652b24d33c1ea2b5e7a6 |
Close
Hashes for curated_tokenizers-0.9.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c4113811d05d8d0dbe4ba1c793d1034ede5dfb9cb466b8f736741072d916af6 |
|
MD5 | 6cf16c209369fca9e49e14b1a7b9ceba |
|
BLAKE2b-256 | 9fd2a4c96504408694f15c35b88a91d082a5ad1066d2559e72bab0d3adebfbcf |