Lightweight piece tokenization library
Project description
🥢 Curated Tokenizers
This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:
Tokenizer | Binding | Example model |
---|---|---|
BPE | sentencepiece | |
Byte BPE | Native | RoBERTa/GPT-2 |
Unigram | sentencepiece | XLM-RoBERTa |
Wordpiece | Native | BERT |
⚠️ Warning: experimental package
This package is experimental and it is likely that the APIs will change in incompatible ways.
⏳ Install
Curated tokenizers is availble through PyPI:
pip install curated_tokenizers
🚀 Quickstart
The best way to get started with curated tokenizers is through the
curated-transformers
library. curated-transformers
also provides functionality to load tokenization
models from Huggingface Hub.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Close
Hashes for curated_tokenizers-0.0.8-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 56bc82bb3ccac138b5da6df5bf785522fa888a08ae521b650493d5e60d277855 |
|
MD5 | 9cf2bceca88e2cc355409a14db4eeed4 |
|
BLAKE2b-256 | 3857e045a84a8b14dbb7617b21645a7b2f8ec9b53d7c65b44036387bfe09deb9 |
Close
Hashes for curated_tokenizers-0.0.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af42fffeab7eaf8e2e03d52f17bdeb2d00d864685bc7faca242f14c2814380fd |
|
MD5 | 58bf5e651fa9f9a935ccbd69300fb28d |
|
BLAKE2b-256 | a4fccfcf511db5658dcf1c8822de846ae97c1c955fb633b5e2d6d36b1f0fa441 |
Close
Hashes for curated_tokenizers-0.0.8-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 887e3e92d564a9a3beac7ded2e633aa17d5754181935fb5b3bf019275247f9df |
|
MD5 | 45e7ee567639fb6b4918aa1a841cc6b8 |
|
BLAKE2b-256 | aa143ba02398e29d2639e33c7f61b077e74c82e0c2b1b646d33207d6a7f2c96f |
Close
Hashes for curated_tokenizers-0.0.8-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 247ff8998198b16ea1d9419c6655e6e25b9cf3cf6a4db1ebed0beebb5e69e1eb |
|
MD5 | 7162b4ae2d33b6c6082f15fab229e0ed |
|
BLAKE2b-256 | 8dd404f2e5a29661ee29df8ed263cf099dfe19607da0af122bae8b711e49c052 |
Close
Hashes for curated_tokenizers-0.0.8-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b26ad4c62f00778f94266e8d2aac4fad25fdd915c2d0bf4b993be10dd9cdf27e |
|
MD5 | 2b76e493b278094dad959793f7ba2dc5 |
|
BLAKE2b-256 | d6c8a2ea60bdd40d8865110a7b4c2e855890cfcdb5d7adbadef60ba41b73b1bf |
Close
Hashes for curated_tokenizers-0.0.8-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5797c513f1d9188c8fbd8b38b0f863561e0ae2858f9a4269dc538e6100bae833 |
|
MD5 | acb7749b747e78db4ed8ab39a6a10116 |
|
BLAKE2b-256 | 91e3e1fe28b8c9c5a7c3563a4ed806f9162ae650f122494cfff59d59a575df66 |
Close
Hashes for curated_tokenizers-0.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 39bc2b36cd40a2583541b54e67ae8eaf45331ca6adb086c2feb7d14da6dd8f14 |
|
MD5 | 9f439d0ddc8924194476c9a93c133eac |
|
BLAKE2b-256 | c4b15839e407714c368a4e0cf55980bf075965d8988c87531c83219f097776ec |
Close
Hashes for curated_tokenizers-0.0.8-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 122400061351a61f4608acaf2bfa99da2354ad950626aaa3b7f429e2bf6401ca |
|
MD5 | 7818f2331bd90707599bdff81ecd663a |
|
BLAKE2b-256 | 7a618ddaff6d3b0837ae78675dc192aa3c7e6381618e6ab8118985b747b2aa22 |
Close
Hashes for curated_tokenizers-0.0.8-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | db635ac5d7b5c266dfc3779effe2fa1741cd8a1cd464418fbbb5434a3697bf77 |
|
MD5 | 894827d430e270e3706301eea6cec20e |
|
BLAKE2b-256 | 66586fe6cbf9e9d1fd656d58d10a8aa8a293f512e91ed64d32af2b13c984c842 |
Close
Hashes for curated_tokenizers-0.0.8-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0480cbad4ac6cd25d28bcf6ac387514728dae0c2bc4b21fc7c6df240ac48e5d0 |
|
MD5 | 1b8c87701cbd389a7c270b0109b4926f |
|
BLAKE2b-256 | d4e6621c2f30f675382a625c86a950ed45d4894163f17b607e5ff92e96142c9a |
Close
Hashes for curated_tokenizers-0.0.8-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 460cd6a533d8ad6c093cc442c8c4b9836b88e4f2e069c0a7423a74b28d47623f |
|
MD5 | a0dab3d96175935f93a48cf86ed02d17 |
|
BLAKE2b-256 | 735063d505156b35466cec019c16fbc8e52240f663fbaa073cfa21f8cd4929dc |
Close
Hashes for curated_tokenizers-0.0.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bf33a34c472abe89724b61708956f00f4b567a8f3ec6b4268e81e2f73aa1440d |
|
MD5 | b4ba721819534854d727c044de1f414c |
|
BLAKE2b-256 | b7c7f9478b911fd3f01b6c50bbbdf699fd4f54dfd470c78176353e979f83e993 |
Close
Hashes for curated_tokenizers-0.0.8-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59cfccbbd93841b8d20fc6ad50463ec7724ae49d414e696f93694a5a8005a4d9 |
|
MD5 | ee1f4c352145de8b0e97e8ac41f4eac5 |
|
BLAKE2b-256 | 526290bc1b6570cea2f0b73575a27a6ad9a4915382dcf5c3474cc693edb0ad37 |
Close
Hashes for curated_tokenizers-0.0.8-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49911eeca4b8d4543c11cb020707b0bb33f4541f2e64c53796ab09da0321e16c |
|
MD5 | 6b6f98f67ac00a2e26a8df90065e1a68 |
|
BLAKE2b-256 | 9e23f89412cb5b9c82e1c61979260b2e0ad9c220e313e0ab3b4885ec2415a212 |
Close
Hashes for curated_tokenizers-0.0.8-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09aff37f1b1ecd95e02c898c5756a48d650c4c527c402d69b0e69d3b362e8467 |
|
MD5 | e2e770138d7fe481eaeb695cb19c258f |
|
BLAKE2b-256 | e3da430c17ac5be2ed14084eff8ccb47a1bff154ebaecef24af1fa52c531e595 |
Close
Hashes for curated_tokenizers-0.0.8-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbd8322dada11dd3e8020523cf6413d7a839e76d7cba6d202bf41a92a99f0063 |
|
MD5 | 83fd0ec623bdffb7a9fa2823887b4f31 |
|
BLAKE2b-256 | 0dbfd902d819af1647efe95818c32a3d49d912af47e2533da9db0d9a479b67c2 |
Close
Hashes for curated_tokenizers-0.0.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | faca4ed350ed19fd706afaf1e3f3ac0a5a83067c9087638b3364a4f9cdca6e2f |
|
MD5 | a5427c9be0daf9dec6b735bbc383af98 |
|
BLAKE2b-256 | 15af5b4307967f1aa39e9290509adf20f6c3d91e7424772d26ed9a7b49dc56e4 |
Close
Hashes for curated_tokenizers-0.0.8-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29d484c5a14d5856bcae7e211f2d0d5b99bb7366a8215adf00812612afb88e1f |
|
MD5 | 85e98dc4bf07c688677cf7a714d94ef9 |
|
BLAKE2b-256 | 2c31d45906a04d9e642798bcfe93d33695a0f799ddaf64add8be6616568012ca |
Close
Hashes for curated_tokenizers-0.0.8-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 32268ce4ed90faed38ee95a22ab6d484576010ce7d65dc945d33e774e6f0901d |
|
MD5 | 975f127b5c0579749fe6d8e940232b8e |
|
BLAKE2b-256 | 6f0bead29e02f9cb722c1a2a469092a497c513d2821dc57512aeea8397c99e4f |
Close
Hashes for curated_tokenizers-0.0.8-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c72f3028585f357cc0ed785f4a3bdcef5507ee97e9fc3227bb12dee604fcae46 |
|
MD5 | 9211b77a59b5e822cbd849e32395932f |
|
BLAKE2b-256 | 0bf522519c4a1e1d81a0113f3ae43123c1e953eb2f84589b32167742bd18fc7f |
Close
Hashes for curated_tokenizers-0.0.8-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6788272cc0ff6edfd2124891b0131a024058715493c52435571c994cd624f7ad |
|
MD5 | df9e0f90976e23b265041b706598050b |
|
BLAKE2b-256 | e8602ee4f6b9bbf710ce3984f6783a5c15c0a57d26657e34146db96f9d3abbe9 |
Close
Hashes for curated_tokenizers-0.0.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22a123fddd16721ac799e29840409566550bb927d6acc8ae14513932c714a170 |
|
MD5 | 51dacf48bad9608f9e0c2e6129de04da |
|
BLAKE2b-256 | a5a12d38edf65ac70781ae2da38f0e7c8fbfa628420bd4c35088d12bb9c6705e |
Close
Hashes for curated_tokenizers-0.0.8-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2cc6f48569609d8a1a01dbdd8e430368b3ccda078e0959f6dba5bcae6bfdb564 |
|
MD5 | 3c2d6023c2c59f5535a8dd35dfb24b86 |
|
BLAKE2b-256 | 456f1ea16ff3d4c292fadf66219c191d9d74fa528b4e82bed0dd30629bee8209 |
Close
Hashes for curated_tokenizers-0.0.8-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 076e7874edf453657173cacaa8f4cf82e928c16021839145dca127295695b3b5 |
|
MD5 | 888148b6a2283c5f6c3bc8627b9e8203 |
|
BLAKE2b-256 | 0dea544a75a4d0f839e5ed4649fe627bce6cc7bd7281b9da17d6375e60877050 |
Close
Hashes for curated_tokenizers-0.0.8-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87cf6efd1bc260fb51cc79065a39420d1a0b8c39265da4916a9f17ec8701f786 |
|
MD5 | 868a1fa1a1bd8544ed741b6a8de413be |
|
BLAKE2b-256 | 581939d42b89bdb2d16831e57c57895d86c51af0cf0867db12dd3bc1cbdd18d0 |
Close
Hashes for curated_tokenizers-0.0.8-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 08c601b4fc3e3ff6023b2a259d5ed6a2cb3efb34bda44368051544c7395db037 |
|
MD5 | 37228a9282b0b4b8e5c97c320d42bcb1 |
|
BLAKE2b-256 | 4c5a6dfb996b3c7c0006279db75d87c9836449652b252bc703cbd073c6db43cf |
Close
Hashes for curated_tokenizers-0.0.8-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f5d1d13a28eb043cdd1f527875d0b98e0d71e84df9790c35d20d62ba869e21ed |
|
MD5 | e2060848ae4f1e061b3ae45bb1284a14 |
|
BLAKE2b-256 | b36f5452507b9f75ae89c933baa7b2b705c262f3f9bdb032cae7e353570e1e9c |