Lightweight piece tokenization library
Project description
🥢 Curated Tokenizers
This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:
Tokenizer | Binding | Example model |
---|---|---|
BPE | sentencepiece | |
Byte BPE | Native | RoBERTa/GPT-2 |
Unigram | sentencepiece | XLM-RoBERTa |
Wordpiece | Native | BERT |
⚠️ Warning: experimental package
This package is experimental and it is likely that the APIs will change in incompatible ways.
⏳ Install
Curated tokenizers is availble through PyPI:
pip install curated_tokenizers
🚀 Quickstart
The best way to get started with curated tokenizers is through the
curated-transformers
library. curated-transformers
also provides functionality to load tokenization
models from Huggingface Hub.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Close
Hashes for curated_tokenizers-0.0.7-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b2a54518b94d9733e6780574f004041d217f984a8f4280f3e602feac08056c6e |
|
MD5 | 2f69d08dbb6bfd63c9f08d14c48e7ad0 |
|
BLAKE2b-256 | c52588ac587b0718ebf5b8e94432ea02d8304ddd72f014df3406b54ffd2305d1 |
Close
Hashes for curated_tokenizers-0.0.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f5a3d8b0ad2fc44f45d62834bed44849cd7a7ef5879844af0526d5c6c7026d2 |
|
MD5 | 8d8006c44bce9065eacd7d559540b0a6 |
|
BLAKE2b-256 | 63ae5a892033ddcbd0f2c072f24662daba8fee8f772047339160f0cf3389a93c |
Close
Hashes for curated_tokenizers-0.0.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6cac8eea50e9bfc55f63d3517ed8b79149c27a559a3c5bcc2f811240920b2c1 |
|
MD5 | 1951c488190a62ee77605b7a65287462 |
|
BLAKE2b-256 | 5dd6d4cb39a2bca58a332b1074508f8f3961b1be85c6c048a41b4b02dd95aef3 |
Close
Hashes for curated_tokenizers-0.0.7-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a2833e2b4e18d112b2f36c97a6d07a6c4ce45ad5befe51ebff30f05ed436610 |
|
MD5 | 9f9af5a8240b8f673d0f5f83167c82ca |
|
BLAKE2b-256 | 93b138f7f527a22a9a2e9393f52dc875b7875784b644bfcd5c7703882b2d1c36 |
Close
Hashes for curated_tokenizers-0.0.7-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 345ad774d3ac8d60d4859b0c3f96ed7cd7a8404c855760d8808f8cecaab7f12e |
|
MD5 | 1760d76ae2fd811739b4a596b8075b2b |
|
BLAKE2b-256 | 91819709ed42e751148c7853feb54cacba7295072eebde2dc5c003331504976f |
Close
Hashes for curated_tokenizers-0.0.7-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f82e14acfbc16bb4b4303489e561522b37b7e6ce741cee798529ba51eeda1476 |
|
MD5 | cc407eb893416cb05138849cf0971e7a |
|
BLAKE2b-256 | 5834a87bff5c21ac46b0621a86aea39696a3f489b4fd35552c571a1620655832 |
Close
Hashes for curated_tokenizers-0.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 37000c1ebbd4639b18477292e6a03312a25ccd1003b3d74db51af7b93bc1bd65 |
|
MD5 | 591549eb2c1b30c5c44933c6ab89f1ff |
|
BLAKE2b-256 | a699c717badc07bea0c347867fb3d8631a9f8099bd8c1294344adf94c6f745db |
Close
Hashes for curated_tokenizers-0.0.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 448f897914cc285c2512a98dbf41eef937ef15c21cef885828d8b6efc6907b61 |
|
MD5 | c210d0187190c7afc9c5c85d64d77786 |
|
BLAKE2b-256 | 2ef3a74baca025795e9fafcceb277a26fe000d5706a51a21a0784e742622720b |
Close
Hashes for curated_tokenizers-0.0.7-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14e53934c3bd18ff809b1aa6119fdb11831e90d395203edbe1c21954bb817e01 |
|
MD5 | 7ba99e2bfc2a3a5d70c9a7139f34ac3a |
|
BLAKE2b-256 | 8b4448a6b56ee6d96dab6af01319b4bb0b999ae5dd79e314f124919ee5530a14 |
Close
Hashes for curated_tokenizers-0.0.7-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a77121b653c110b5e861b407426b9af55ffd2890e9adeb7bb75ae97c3fdd0c6 |
|
MD5 | 21e16133a5e80271a5aaeb3ea9af6ae7 |
|
BLAKE2b-256 | ab03154ea07588145b1faf5ab9f74ec9e80dea57e32920bb8cbd4d11a1029784 |
Close
Hashes for curated_tokenizers-0.0.7-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6dd63e0acb3c144fc74830ef2a950d94e574e7ecbf8472de916f43313515283a |
|
MD5 | b59973ba68daea87da2fedbb405cfbca |
|
BLAKE2b-256 | 31d7d036791616d0bc89eb81de5d46ffa9b34e1333e6c420d8d28050fbdf1b82 |
Close
Hashes for curated_tokenizers-0.0.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b9c8e89409ac46f92dd015e95cda4efab25dafaefbc834d17f338677826d80bd |
|
MD5 | 3df7440b3ea0cf0cb348c6ee4fc79cc0 |
|
BLAKE2b-256 | 497b86edf0a051de18bdcc07bedb425b9e3c999a927b4a7c47eff5caaf0e2231 |
Close
Hashes for curated_tokenizers-0.0.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d993458d0c48f187d026cfc215a9daee7af5fc33838305f5557bb04e430d8554 |
|
MD5 | 2547b8d2ae78bcf610cb7773c5316b1e |
|
BLAKE2b-256 | 1f79cbddbb037335a5b1dea9eaf4a34d0380d7a12f59ef905219a2a9edee72ff |
Close
Hashes for curated_tokenizers-0.0.7-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a8072c0cf51de4f51fa4b6d65ada5dd04ded4be0a1ac0531a4be868298ef29f |
|
MD5 | 258a391a31e409c15bbae5a8af4c2d69 |
|
BLAKE2b-256 | e22cac0b4984c513dc37ff8fd25bbdcc0de11351be0c421ce65f6a677003e45c |
Close
Hashes for curated_tokenizers-0.0.7-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93fd7401e03f63940090319077e74f1bf913ba2d8d605439414ff123564811a9 |
|
MD5 | f7f0620584f361d61da75a5986adbb63 |
|
BLAKE2b-256 | 6104ced9b45557896758e66b93199277eb122f855a69e569b0b2abfc0cf0980e |
Close
Hashes for curated_tokenizers-0.0.7-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 28485ad0f9fe1a45f9feb513e093c43a4b77202fb4246abc24c6453c65388db7 |
|
MD5 | 3a43e1a46b1e60285bbd771ea6d756ee |
|
BLAKE2b-256 | cffa2c84864cf891d5c43bca09ba20b4df6b8f56cd06e8e8a36c9e661f09d7f1 |
Close
Hashes for curated_tokenizers-0.0.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a873cf3ac5456761b3f05afeb5e75a091cd13c3a8a022d98465fa229aa20b6d |
|
MD5 | dbc5e0c3edf2b4affefa2424bd9e5461 |
|
BLAKE2b-256 | 2d2eea5716f79bcb4aff8e436e11119398af9c3a1d7a4612190dfc40e0cdb9a7 |
Close
Hashes for curated_tokenizers-0.0.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38c21dc6d7440ae0304ce5ba91bb970d1fb364179c0c5015898cfd653e8b04dc |
|
MD5 | 6941f74c1834e98b4541ce158f303816 |
|
BLAKE2b-256 | 0f98fd0eacc8238aa4450a7afec74dd41026c269dc112242f7be4ece119d6dcb |
Close
Hashes for curated_tokenizers-0.0.7-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4045ebb2c38054d92b5cfeb2a5a06eb569c06c840a63bec8a2b90ada043eb275 |
|
MD5 | 94ab31ae1e013be6614e9a86469662de |
|
BLAKE2b-256 | c94dbce24fbafaf469779ebdf97c44cfc2bc8273a9b5149b0dd3862ecaea3004 |
Close
Hashes for curated_tokenizers-0.0.7-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c3998e79c0626ab9c8d164dd401ab5c761a32f46e402a84763182451d42d6701 |
|
MD5 | 7f84bccf04c912b5fcc568e70bc1a9c7 |
|
BLAKE2b-256 | 307e43c36efdb3d4c6933d04ff739bad7624647e1b5ef0150d2502affe3570d9 |
Close
Hashes for curated_tokenizers-0.0.7-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 51aefe3b423433dc8f91b9a640458d8ece6d4b35b07226c1396907d7021f70b9 |
|
MD5 | 1aad87606b28d49f286ca1315671597a |
|
BLAKE2b-256 | 4f37271d4c2da6726e052b3c8f796f8f62a35d737683e2eb380ab08876af72df |
Close
Hashes for curated_tokenizers-0.0.7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00847b3c6383a11c15a7b9acc35759134318cb2bf487997dfa04678afe0ddef1 |
|
MD5 | c6a30a143cfaddef87018027f166a57c |
|
BLAKE2b-256 | c0ad6cb1d029551589851126e8f875879be2a229ceec87360e26bcc70ce0277c |
Close
Hashes for curated_tokenizers-0.0.7-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67a1f1a3bd4a2d0f9944ef6f8c1e692fae444d1472bde7277be6a51ad31eb1e9 |
|
MD5 | 77c6f0fdb356cb68df9cf27519a94bd4 |
|
BLAKE2b-256 | eaa9fd09bacc1d268b7ce65e0d76e2db7bb4cffe8f8738b9a6208cbcde7f27ab |
Close
Hashes for curated_tokenizers-0.0.7-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 50009c546ee2ebcb07de61ccc58165f5bd957ec17a2edfa5fbc89ff1d75f7891 |
|
MD5 | 493114b99d26f5e469756badb34c036a |
|
BLAKE2b-256 | 30a58fec572adfb1bb1bd1b1da5637da8704c6c6424227078ca13cb20040474d |
Close
Hashes for curated_tokenizers-0.0.7-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e862bb62b2555a62318eff0f7a5f5ca23ae68413f3dbad7f209414c169862d77 |
|
MD5 | 189abb54787e829db687a15a56b2fed4 |
|
BLAKE2b-256 | 2b9a6cce938f8ecd03dbf42f8dfda93ac9d792fdac5de501ca28034706c77d30 |
Close
Hashes for curated_tokenizers-0.0.7-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 582819b45a6403304b4e2f2b6035fce6dde4285017a5b90799c3839d0fc3b645 |
|
MD5 | 7366d8d40fb06e0b83ff3088d4b8d698 |
|
BLAKE2b-256 | 537e97e73e70c1beda5e743fed345aa8556234d701ce292a0b5e6453aedf8da2 |
Close
Hashes for curated_tokenizers-0.0.7-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b9af84ff19569fc9b71b3afeb778154322d495381e5e24ed8d6c9fc8afa1423 |
|
MD5 | 247183636cee58978af00c41000956fe |
|
BLAKE2b-256 | c3bfeecad1bf5195cb686deac249b82408519a205b59467193ceb46652d9cf21 |