🍴 cutlery
This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:
| Tokenizer | Binding | Example model |
|---|---|---|
| BPE | sentencepiece | |
| Byte BPE | Native | RoBERTa/GPT-2 |
| Unigram | sentencepiece | XLM-RoBERTa |
| Wordpiece | Native | BERT |
⚠️ Warning: experimental package
This package is experimental and it is likely that the APIs will change in incompatible ways.
⏳ Install
Cutlery is availble through PyPI:
pip install cutlery
🚀 Quickstart
The best way to get started with cutlery is through the
curated-transformers
library. curated-transformers also provides functionality to load tokenization
models from Huggingface Hub.
1 maintainer
Release files for cutlery 0.0.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| cutlery-0.0.6.tar.gz | 2.2 MB | Details |
Built distributions (wheels)
Total release size: 20.2 MB
Release files / cutlery-0.0.6.tar.gz
| Download URL | cutlery-0.0.6.tar.gz |
|---|---|
| Size | 2.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8931662d3cd32ccf0242e8aa001f8f82461b8933fe06f3e9abf4ef3a2bd8b2d0
|
|
BLAKE2b-256 checksum How to use checksums |
123b9dcef9d6efcdb666367356ad3cd464b69fa491c87a737d4600eb3ac21f9a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp311-cp311-win_amd64.whl
| Download URL | cutlery-0.0.6-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 598.5 kB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
009577c2846e232a6adbf0c9b53f74486d4bdae1c19bc05b0fd0b461544dc247
|
|
BLAKE2b-256 checksum How to use checksums |
fde701d4aaa7b48b4dd8b322982831d9995a2293fb2cb1497e2eedf2c53c2aee
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | cutlery-0.0.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 687.6 kB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
70429c612e319a4da8ebe3c168f3cf716cb1a3eec57649a068ad7b0a751a0cf4
|
|
BLAKE2b-256 checksum How to use checksums |
07ea038288ac5c15aba0f2a81b09e002440a8668b9f45539020fd69632fa705f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | cutlery-0.0.6-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 663.5 kB |
| Tags | CPython 3.11 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
13949c5b3ae837d36983f9363513ba7ed4c392527a18b944d0ba6bac5df2a5a9
|
|
BLAKE2b-256 checksum How to use checksums |
b4fabbf5b25f1bed1968eaa37461205efc981c4b10f71e0cdfbd5d68f09f917e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp311-cp311-macosx_11_0_arm64.whl
| Download URL | cutlery-0.0.6-cp311-cp311-macosx_11_0_arm64.whl |
|---|---|
| Size | 662.5 kB |
| Tags | CPython 3.11 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
31a548868649787394f69793fcaa5b42589790fb82a3e2583fd93806827b1c6a
|
|
BLAKE2b-256 checksum How to use checksums |
cfbac1b362e07319e2763ac27ff5958f2a3c979f4f8409c2fdf513882c726bbf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp311-cp311-macosx_10_9_x86_64.whl
| Download URL | cutlery-0.0.6-cp311-cp311-macosx_10_9_x86_64.whl |
|---|---|
| Size | 693.1 kB |
| Tags | CPython 3.11 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
1f131750a6aae3dc2a27aa8e4f07236082c81566d82cb99482972b211595ae8f
|
|
BLAKE2b-256 checksum How to use checksums |
de0010fd7c0155512749e07755340dcb5d62fca9196406dc90d861fc07b00d2a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp310-cp310-win_amd64.whl
| Download URL | cutlery-0.0.6-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 600.3 kB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
937b9392a4dde11a133c7ed20dbead8bf7e6957f41953d5070727a41da713948
|
|
BLAKE2b-256 checksum How to use checksums |
009f47dd791c039470609ce23a7729f47d5851694157e97d072bd2deb1de3633
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | cutlery-0.0.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 687.8 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
d807f5744525c51738aa47d166c2456cf63eeccacf0de9a97502dde2d524cc2f
|
|
BLAKE2b-256 checksum How to use checksums |
85dc6fa6b33bf6098360b9fca5afb5dbeb009df77321c861653f7b2b7b72a9ff
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | cutlery-0.0.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 663.4 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
e9d91c3c145056c4ebb236773ec578b656d2fe2e48ba215bef4eb43d34b3cd21
|
|
BLAKE2b-256 checksum How to use checksums |
fec9caf59c3f04e488a220e516aabf78226ee08059d1b5c22a6300afba484213
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp310-cp310-macosx_11_0_arm64.whl
| Download URL | cutlery-0.0.6-cp310-cp310-macosx_11_0_arm64.whl |
|---|---|
| Size | 665.5 kB |
| Tags | CPython 3.10 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
164e0a1406b2099d38121a02e5bfaee56ace1a739f4976d43f857b6fc05c63c2
|
|
BLAKE2b-256 checksum How to use checksums |
6491824b3b8ba0ea8e88c81c5caf871459d0a1c8869f462686b5ce599911c945
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp310-cp310-macosx_10_9_x86_64.whl
| Download URL | cutlery-0.0.6-cp310-cp310-macosx_10_9_x86_64.whl |
|---|---|
| Size | 696.1 kB |
| Tags | CPython 3.10 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
4301df4b13ea04723c2b198c4559d137c837b5691fb5c6f6178596bea1ca8e7f
|
|
BLAKE2b-256 checksum How to use checksums |
bc02702eedcfb466272059c5e5e10f34a98d92f493d46c290839525ce0858cd9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp39-cp39-win_amd64.whl
| Download URL | cutlery-0.0.6-cp39-cp39-win_amd64.whl |
|---|---|
| Size | 602.3 kB |
| Tags | CPython 3.9 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
ae2638ff53f977a3c6f5533e1907faea24e251ff42432cb3e88dd62e2b03baca
|
|
BLAKE2b-256 checksum How to use checksums |
8b728893b339326e84a81a6c91e7bc86d37cd0d0d9655bc61a9b0d90d57d074c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | cutlery-0.0.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 690.8 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
370354a458913f5a7b66e0cbb0a21a18f27eadd2ff0867a920d540d34651fff2
|
|
BLAKE2b-256 checksum How to use checksums |
825c8f6c502eaee85d545d0944a89b8d0e37581b21815f044a7a20f15de33280
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | cutlery-0.0.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 666.8 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
c004ff2c9d3e035df3c33e0e40586ed7392e3e59085088069d5638103eb8d253
|
|
BLAKE2b-256 checksum How to use checksums |
36ebc3357427f567095c2f319352463e106cb2c1aa674f71a377733524c70eb7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp39-cp39-macosx_11_0_arm64.whl
| Download URL | cutlery-0.0.6-cp39-cp39-macosx_11_0_arm64.whl |
|---|---|
| Size | 665.5 kB |
| Tags | CPython 3.9 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
4f219cf5cf457f8f91f60973b6d181c2d916bded235592971a607649d145da03
|
|
BLAKE2b-256 checksum How to use checksums |
6dde3994e96fc28a055439f79d38a051601e3f8bf3106813586de9fc8f1ed705
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp39-cp39-macosx_10_9_x86_64.whl
| Download URL | cutlery-0.0.6-cp39-cp39-macosx_10_9_x86_64.whl |
|---|---|
| Size | 696.3 kB |
| Tags | CPython 3.9 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
eb330d55cadd5f89326ab552b7178ef8867aa7a783778cd324d6fd90876b481c
|
|
BLAKE2b-256 checksum How to use checksums |
d96d517558fe777a56d62167477acd93942c68585edf69bad6583456c28501e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp38-cp38-win_amd64.whl
| Download URL | cutlery-0.0.6-cp38-cp38-win_amd64.whl |
|---|---|
| Size | 602.6 kB |
| Tags | CPython 3.8 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
0aa6d1940f357a0e2cb12ae80351a1b139db3ac319a4d2199d13f3f9ad70e615
|
|
BLAKE2b-256 checksum How to use checksums |
03fd1bbbb27b3cc36515854d532271553a48c15592b550b757276cd47159b981
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | cutlery-0.0.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 692.0 kB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
94223a0496418b288f60e3a3b1144ae149ce91d9feac2dd6e9214a34dc9394fb
|
|
BLAKE2b-256 checksum How to use checksums |
a87b25be33229ad3dba58ada84b56f5d09a0c4462630ee8fe3af06978fb55ba6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | cutlery-0.0.6-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 668.2 kB |
| Tags | CPython 3.8 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
749d9a9f3feb92cff48ea351f264947292bd8dd0f0f3e3c9ec5eeeeef2ce3b2b
|
|
BLAKE2b-256 checksum How to use checksums |
015d84cf282fd15374228dcc7b3c0e2a80650b11c3f9916e1f1dffc90136b08e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp38-cp38-macosx_11_0_arm64.whl
| Download URL | cutlery-0.0.6-cp38-cp38-macosx_11_0_arm64.whl |
|---|---|
| Size | 663.1 kB |
| Tags | CPython 3.8 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
56a833cfee4d390f3404dcaf3b9888ec3d64d28cc0a9dc3e182c0ab206bdc791
|
|
BLAKE2b-256 checksum How to use checksums |
633e902b236b5cd1028f5de3d858447eec153fc6c20c8c91f2bfce7946820251
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp38-cp38-macosx_10_9_x86_64.whl
| Download URL | cutlery-0.0.6-cp38-cp38-macosx_10_9_x86_64.whl |
|---|---|
| Size | 693.5 kB |
| Tags | CPython 3.8 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
8e18a5cc50a762a406b025c9d8d95f223be6f59890e93b389d40b2793b508111
|
|
BLAKE2b-256 checksum How to use checksums |
616654251e2e59d9bee10360719b2eac10f347be59d14fff006f25742a8c01c8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp37-cp37m-win_amd64.whl
| Download URL | cutlery-0.0.6-cp37-cp37m-win_amd64.whl |
|---|---|
| Size | 601.1 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Windows x86-64 |
|
SHA-256 checksum How to use checksums |
df66e937d8acfd32d7cd6c0c02ce35451d2cfa008b09ca1dbef4e7d7ff704889
|
|
BLAKE2b-256 checksum How to use checksums |
67e87a679819c3b9468b607b522ea8370680954ea8872342c488ccc2983c364e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | cutlery-0.0.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 688.8 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
d5730518ce54207ee6f258c24badf4750d882f2a9b28b39e0644324d8deac374
|
|
BLAKE2b-256 checksum How to use checksums |
8e042dfa34d5f396d36e3fa41b545bdd53f32bb5531550a49cc6ec08a75891c4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | cutlery-0.0.6-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 665.8 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
95025b2c6e985c2dcf068df971e6273b7d10dc43a23759506134745372c889d9
|
|
BLAKE2b-256 checksum How to use checksums |
40bb560ac1262b87a898b4101c7432a395974e96248391fcc29422756840c3ef
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp37-cp37m-macosx_10_9_x86_64.whl
| Download URL | cutlery-0.0.6-cp37-cp37m-macosx_10_9_x86_64.whl |
|---|---|
| Size | 692.8 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
8470c887d39970f03745916687fba7855b1b4e9c34a94dc9b22a2cb97c7e5cc3
|
|
BLAKE2b-256 checksum How to use checksums |
9bb8dd0db9171007b4323bcdb33c698de6c6702b627abf3bb03eaa7f43db9d14
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp36-cp36m-win_amd64.whl
| Download URL | cutlery-0.0.6-cp36-cp36m-win_amd64.whl |
|---|---|
| Size | 653.4 kB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Windows x86-64 |
|
SHA-256 checksum How to use checksums |
a1af4675a1839f124f00086d0caf651c08200431ab4c806060bf4d066a66092a
|
|
BLAKE2b-256 checksum How to use checksums |
a76e18698269f13ee289a4b459629780d2cd196ca24600af62ac20a21759a2b2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | cutlery-0.0.6-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 688.7 kB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
2feeb6c1f824ad2aaee81956efd43ff14c92ca3bf08d4c3bd1eaaca1de8dbd5d
|
|
BLAKE2b-256 checksum How to use checksums |
33b25307b491bea1eb1cecefd9061905932157b72eafdffc9d2626b9a778c64d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
Release files / cutlery-0.0.6-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | cutlery-0.0.6-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 665.9 kB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
934ccabb8fb4db03c7b564c9c9a9bbff3fa29b3e41a2952205512ef547bc352c
|
|
BLAKE2b-256 checksum How to use checksums |
c909ba79534f3c48b2bd593239286f7d43ffd1aeb45571c136312591b3199e32
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.7.9
|
