Skip to main content

Lightweight piece tokenization library

Project description

🥢 Curated Tokenizers

This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:

Tokenizer Binding Example model
BPE sentencepiece
Byte BPE Native RoBERTa/GPT-2
Unigram sentencepiece XLM-RoBERTa
Wordpiece Native BERT

⚠️ Warning: experimental package

This package is experimental and it is likely that the APIs will change in incompatible ways.

⏳ Install

Curated tokenizers is availble through PyPI:

pip install curated_tokenizers

🚀 Quickstart

The best way to get started with curated tokenizers is through the curated-transformers library. curated-transformers also provides functionality to load tokenization models from Huggingface Hub.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

curated-tokenizers-0.0.8.tar.gz (2.2 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

curated_tokenizers-0.0.8-cp311-cp311-win_amd64.whl (604.5 kB view details)

Uploaded CPython 3.11Windows x86-64

curated_tokenizers-0.0.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (692.3 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

curated_tokenizers-0.0.8-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (669.0 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ ARM64

curated_tokenizers-0.0.8-cp311-cp311-macosx_11_0_arm64.whl (665.5 kB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

curated_tokenizers-0.0.8-cp311-cp311-macosx_10_9_x86_64.whl (696.9 kB view details)

Uploaded CPython 3.11macOS 10.9+ x86-64

curated_tokenizers-0.0.8-cp310-cp310-win_amd64.whl (606.4 kB view details)

Uploaded CPython 3.10Windows x86-64

curated_tokenizers-0.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (692.4 kB view details)

Uploaded CPython 3.10manylinux: glibc 2.17+ x86-64

curated_tokenizers-0.0.8-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (668.5 kB view details)

Uploaded CPython 3.10manylinux: glibc 2.17+ ARM64

curated_tokenizers-0.0.8-cp310-cp310-macosx_11_0_arm64.whl (668.8 kB view details)

Uploaded CPython 3.10macOS 11.0+ ARM64

curated_tokenizers-0.0.8-cp310-cp310-macosx_10_9_x86_64.whl (700.0 kB view details)

Uploaded CPython 3.10macOS 10.9+ x86-64

curated_tokenizers-0.0.8-cp39-cp39-win_amd64.whl (608.6 kB view details)

Uploaded CPython 3.9Windows x86-64

curated_tokenizers-0.0.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (696.5 kB view details)

Uploaded CPython 3.9manylinux: glibc 2.17+ x86-64

curated_tokenizers-0.0.8-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (673.3 kB view details)

Uploaded CPython 3.9manylinux: glibc 2.17+ ARM64

curated_tokenizers-0.0.8-cp39-cp39-macosx_11_0_arm64.whl (668.6 kB view details)

Uploaded CPython 3.9macOS 11.0+ ARM64

curated_tokenizers-0.0.8-cp39-cp39-macosx_10_9_x86_64.whl (700.0 kB view details)

Uploaded CPython 3.9macOS 10.9+ x86-64

curated_tokenizers-0.0.8-cp38-cp38-win_amd64.whl (608.8 kB view details)

Uploaded CPython 3.8Windows x86-64

curated_tokenizers-0.0.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (698.0 kB view details)

Uploaded CPython 3.8manylinux: glibc 2.17+ x86-64

curated_tokenizers-0.0.8-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (673.9 kB view details)

Uploaded CPython 3.8manylinux: glibc 2.17+ ARM64

curated_tokenizers-0.0.8-cp38-cp38-macosx_11_0_arm64.whl (666.1 kB view details)

Uploaded CPython 3.8macOS 11.0+ ARM64

curated_tokenizers-0.0.8-cp38-cp38-macosx_10_9_x86_64.whl (697.1 kB view details)

Uploaded CPython 3.8macOS 10.9+ x86-64

curated_tokenizers-0.0.8-cp37-cp37m-win_amd64.whl (606.9 kB view details)

Uploaded CPython 3.7mWindows x86-64

curated_tokenizers-0.0.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (694.5 kB view details)

Uploaded CPython 3.7mmanylinux: glibc 2.17+ x86-64

curated_tokenizers-0.0.8-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (671.5 kB view details)

Uploaded CPython 3.7mmanylinux: glibc 2.17+ ARM64

curated_tokenizers-0.0.8-cp37-cp37m-macosx_10_9_x86_64.whl (696.3 kB view details)

Uploaded CPython 3.7mmacOS 10.9+ x86-64

curated_tokenizers-0.0.8-cp36-cp36m-win_amd64.whl (658.9 kB view details)

Uploaded CPython 3.6mWindows x86-64

curated_tokenizers-0.0.8-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (694.5 kB view details)

Uploaded CPython 3.6mmanylinux: glibc 2.17+ x86-64

curated_tokenizers-0.0.8-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (670.9 kB view details)

Uploaded CPython 3.6mmanylinux: glibc 2.17+ ARM64

File details

Details for the file curated-tokenizers-0.0.8.tar.gz.

File metadata

  • Download URL: curated-tokenizers-0.0.8.tar.gz
  • Upload date:
  • Size: 2.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.7.9

File hashes

Hashes for curated-tokenizers-0.0.8.tar.gz
Algorithm Hash digest
SHA256 e3e052c72c0adfad919cf6cb88a505d982100425b4b1aaa3f7b3e4cb4a95b818
MD5 d0df097de068660565d45db8c103ee82
BLAKE2b-256 fcc9ef2881a9445a53b38f97222de5a77c483f7e9425800520e48ad2bc40525d

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp311-cp311-win_amd64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 56bc82bb3ccac138b5da6df5bf785522fa888a08ae521b650493d5e60d277855
MD5 9cf2bceca88e2cc355409a14db4eeed4
BLAKE2b-256 3857e045a84a8b14dbb7617b21645a7b2f8ec9b53d7c65b44036387bfe09deb9

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 af42fffeab7eaf8e2e03d52f17bdeb2d00d864685bc7faca242f14c2814380fd
MD5 58bf5e651fa9f9a935ccbd69300fb28d
BLAKE2b-256 a4fccfcf511db5658dcf1c8822de846ae97c1c955fb633b5e2d6d36b1f0fa441

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 887e3e92d564a9a3beac7ded2e633aa17d5754181935fb5b3bf019275247f9df
MD5 45e7ee567639fb6b4918aa1a841cc6b8
BLAKE2b-256 aa143ba02398e29d2639e33c7f61b077e74c82e0c2b1b646d33207d6a7f2c96f

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 247ff8998198b16ea1d9419c6655e6e25b9cf3cf6a4db1ebed0beebb5e69e1eb
MD5 7162b4ae2d33b6c6082f15fab229e0ed
BLAKE2b-256 8dd404f2e5a29661ee29df8ed263cf099dfe19607da0af122bae8b711e49c052

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 b26ad4c62f00778f94266e8d2aac4fad25fdd915c2d0bf4b993be10dd9cdf27e
MD5 2b76e493b278094dad959793f7ba2dc5
BLAKE2b-256 d6c8a2ea60bdd40d8865110a7b4c2e855890cfcdb5d7adbadef60ba41b73b1bf

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp310-cp310-win_amd64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 5797c513f1d9188c8fbd8b38b0f863561e0ae2858f9a4269dc538e6100bae833
MD5 acb7749b747e78db4ed8ab39a6a10116
BLAKE2b-256 91e3e1fe28b8c9c5a7c3563a4ed806f9162ae650f122494cfff59d59a575df66

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 39bc2b36cd40a2583541b54e67ae8eaf45331ca6adb086c2feb7d14da6dd8f14
MD5 9f439d0ddc8924194476c9a93c133eac
BLAKE2b-256 c4b15839e407714c368a4e0cf55980bf075965d8988c87531c83219f097776ec

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 122400061351a61f4608acaf2bfa99da2354ad950626aaa3b7f429e2bf6401ca
MD5 7818f2331bd90707599bdff81ecd663a
BLAKE2b-256 7a618ddaff6d3b0837ae78675dc192aa3c7e6381618e6ab8118985b747b2aa22

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 db635ac5d7b5c266dfc3779effe2fa1741cd8a1cd464418fbbb5434a3697bf77
MD5 894827d430e270e3706301eea6cec20e
BLAKE2b-256 66586fe6cbf9e9d1fd656d58d10a8aa8a293f512e91ed64d32af2b13c984c842

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 0480cbad4ac6cd25d28bcf6ac387514728dae0c2bc4b21fc7c6df240ac48e5d0
MD5 1b8c87701cbd389a7c270b0109b4926f
BLAKE2b-256 d4e6621c2f30f675382a625c86a950ed45d4894163f17b607e5ff92e96142c9a

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp39-cp39-win_amd64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 460cd6a533d8ad6c093cc442c8c4b9836b88e4f2e069c0a7423a74b28d47623f
MD5 a0dab3d96175935f93a48cf86ed02d17
BLAKE2b-256 735063d505156b35466cec019c16fbc8e52240f663fbaa073cfa21f8cd4929dc

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 bf33a34c472abe89724b61708956f00f4b567a8f3ec6b4268e81e2f73aa1440d
MD5 b4ba721819534854d727c044de1f414c
BLAKE2b-256 b7c7f9478b911fd3f01b6c50bbbdf699fd4f54dfd470c78176353e979f83e993

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 59cfccbbd93841b8d20fc6ad50463ec7724ae49d414e696f93694a5a8005a4d9
MD5 ee1f4c352145de8b0e97e8ac41f4eac5
BLAKE2b-256 526290bc1b6570cea2f0b73575a27a6ad9a4915382dcf5c3474cc693edb0ad37

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 49911eeca4b8d4543c11cb020707b0bb33f4541f2e64c53796ab09da0321e16c
MD5 6b6f98f67ac00a2e26a8df90065e1a68
BLAKE2b-256 9e23f89412cb5b9c82e1c61979260b2e0ad9c220e313e0ab3b4885ec2415a212

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 09aff37f1b1ecd95e02c898c5756a48d650c4c527c402d69b0e69d3b362e8467
MD5 e2e770138d7fe481eaeb695cb19c258f
BLAKE2b-256 e3da430c17ac5be2ed14084eff8ccb47a1bff154ebaecef24af1fa52c531e595

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp38-cp38-win_amd64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 dbd8322dada11dd3e8020523cf6413d7a839e76d7cba6d202bf41a92a99f0063
MD5 83fd0ec623bdffb7a9fa2823887b4f31
BLAKE2b-256 0dbfd902d819af1647efe95818c32a3d49d912af47e2533da9db0d9a479b67c2

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 faca4ed350ed19fd706afaf1e3f3ac0a5a83067c9087638b3364a4f9cdca6e2f
MD5 a5427c9be0daf9dec6b735bbc383af98
BLAKE2b-256 15af5b4307967f1aa39e9290509adf20f6c3d91e7424772d26ed9a7b49dc56e4

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 29d484c5a14d5856bcae7e211f2d0d5b99bb7366a8215adf00812612afb88e1f
MD5 85e98dc4bf07c688677cf7a714d94ef9
BLAKE2b-256 2c31d45906a04d9e642798bcfe93d33695a0f799ddaf64add8be6616568012ca

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 32268ce4ed90faed38ee95a22ab6d484576010ce7d65dc945d33e774e6f0901d
MD5 975f127b5c0579749fe6d8e940232b8e
BLAKE2b-256 6f0bead29e02f9cb722c1a2a469092a497c513d2821dc57512aeea8397c99e4f

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 c72f3028585f357cc0ed785f4a3bdcef5507ee97e9fc3227bb12dee604fcae46
MD5 9211b77a59b5e822cbd849e32395932f
BLAKE2b-256 0bf522519c4a1e1d81a0113f3ae43123c1e953eb2f84589b32167742bd18fc7f

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp37-cp37m-win_amd64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 6788272cc0ff6edfd2124891b0131a024058715493c52435571c994cd624f7ad
MD5 df9e0f90976e23b265041b706598050b
BLAKE2b-256 e8602ee4f6b9bbf710ce3984f6783a5c15c0a57d26657e34146db96f9d3abbe9

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 22a123fddd16721ac799e29840409566550bb927d6acc8ae14513932c714a170
MD5 51dacf48bad9608f9e0c2e6129de04da
BLAKE2b-256 a5a12d38edf65ac70781ae2da38f0e7c8fbfa628420bd4c35088d12bb9c6705e

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2cc6f48569609d8a1a01dbdd8e430368b3ccda078e0959f6dba5bcae6bfdb564
MD5 3c2d6023c2c59f5535a8dd35dfb24b86
BLAKE2b-256 456f1ea16ff3d4c292fadf66219c191d9d74fa528b4e82bed0dd30629bee8209

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp37-cp37m-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 076e7874edf453657173cacaa8f4cf82e928c16021839145dca127295695b3b5
MD5 888148b6a2283c5f6c3bc8627b9e8203
BLAKE2b-256 0dea544a75a4d0f839e5ed4649fe627bce6cc7bd7281b9da17d6375e60877050

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp36-cp36m-win_amd64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 87cf6efd1bc260fb51cc79065a39420d1a0b8c39265da4916a9f17ec8701f786
MD5 868a1fa1a1bd8544ed741b6a8de413be
BLAKE2b-256 581939d42b89bdb2d16831e57c57895d86c51af0cf0867db12dd3bc1cbdd18d0

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 08c601b4fc3e3ff6023b2a259d5ed6a2cb3efb34bda44368051544c7395db037
MD5 37228a9282b0b4b8e5c97c320d42bcb1
BLAKE2b-256 4c5a6dfb996b3c7c0006279db75d87c9836449652b252bc703cbd073c6db43cf

See more details on using hashes here.

File details

Details for the file curated_tokenizers-0.0.8-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for curated_tokenizers-0.0.8-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 f5d1d13a28eb043cdd1f527875d0b98e0d71e84df9790c35d20d62ba869e21ed
MD5 e2060848ae4f1e061b3ae45bb1284a14
BLAKE2b-256 b36f5452507b9f75ae89c933baa7b2b705c262f3f9bdb032cae7e353570e1e9c

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page