Lightweight piece tokenization library

These details have not been verified by PyPI

Project links

Homepage

Development Status
- 5 - Production/Stable
Environment
- Console
Intended Audience
- Developers
- Science/Research
License
- OSI Approved :: MIT License
Operating System
- MacOS :: MacOS X
Programming Language
Topic
- Scientific/Engineering

Project description

🍴 cutlery

This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:

Tokenizer	Binding	Example model
BPE	sentencepiece
Byte BPE	Native	RoBERTa/GPT-2
Unigram	sentencepiece	XLM-RoBERTa
Wordpiece	Native	BERT

⚠️ Warning: experimental package

This package is experimental and it is likely that the APIs will change in incompatible ways.

⏳ Install

Cutlery is availble through PyPI:

pip install cutlery

🚀 Quickstart

The best way to get started with cutlery is through the curated-transformers library. curated-transformers also provides functionality to load tokenization models from Huggingface Hub.

Project details

These details have not been verified by PyPI

Project links

Homepage

Development Status
- 5 - Production/Stable
Environment
- Console
Intended Audience
- Developers
- Science/Research
License
- OSI Approved :: MIT License
Operating System
- MacOS :: MacOS X
Programming Language
Topic
- Scientific/Engineering

Release history Release notifications | RSS feed

0.0.6

May 22, 2023

This version

0.0.5

May 19, 2023

0.0.4

May 14, 2023

0.0.3

Oct 18, 2022

0.0.2

Oct 18, 2022

0.0.1

Sep 8, 2022

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cutlery-0.0.5.tar.gz (2.2 MB view hashes)

Uploaded May 19, 2023 Source

Built Distributions

cutlery-0.0.5-cp311-cp311-win_amd64.whl (590.4 kB view hashes)

Uploaded May 19, 2023 CPython 3.11 Windows x86-64

cutlery-0.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (683.1 kB view hashes)

Uploaded May 19, 2023 CPython 3.11 manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (658.6 kB view hashes)

Uploaded May 19, 2023 CPython 3.11 manylinux: glibc 2.17+ ARM64

cutlery-0.0.5-cp311-cp311-macosx_11_0_arm64.whl (658.8 kB view hashes)

Uploaded May 19, 2023 CPython 3.11 macOS 11.0+ ARM64

cutlery-0.0.5-cp311-cp311-macosx_10_9_x86_64.whl (688.5 kB view hashes)

Uploaded May 19, 2023 CPython 3.11 macOS 10.9+ x86-64

cutlery-0.0.5-cp310-cp310-win_amd64.whl (592.2 kB view hashes)

Uploaded May 19, 2023 CPython 3.10 Windows x86-64

cutlery-0.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (683.2 kB view hashes)

Uploaded May 19, 2023 CPython 3.10 manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (658.1 kB view hashes)

Uploaded May 19, 2023 CPython 3.10 manylinux: glibc 2.17+ ARM64

cutlery-0.0.5-cp310-cp310-macosx_11_0_arm64.whl (661.9 kB view hashes)

Uploaded May 19, 2023 CPython 3.10 macOS 11.0+ ARM64

cutlery-0.0.5-cp310-cp310-macosx_10_9_x86_64.whl (691.6 kB view hashes)

Uploaded May 19, 2023 CPython 3.10 macOS 10.9+ x86-64

cutlery-0.0.5-cp39-cp39-win_amd64.whl (594.2 kB view hashes)

Uploaded May 19, 2023 CPython 3.9 Windows x86-64

cutlery-0.0.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (686.1 kB view hashes)

Uploaded May 19, 2023 CPython 3.9 manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (661.6 kB view hashes)

Uploaded May 19, 2023 CPython 3.9 manylinux: glibc 2.17+ ARM64

cutlery-0.0.5-cp39-cp39-macosx_11_0_arm64.whl (661.9 kB view hashes)

Uploaded May 19, 2023 CPython 3.9 macOS 11.0+ ARM64

cutlery-0.0.5-cp39-cp39-macosx_10_9_x86_64.whl (691.8 kB view hashes)

Uploaded May 19, 2023 CPython 3.9 macOS 10.9+ x86-64

cutlery-0.0.5-cp38-cp38-win_amd64.whl (594.4 kB view hashes)

Uploaded May 19, 2023 CPython 3.8 Windows x86-64

cutlery-0.0.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (687.1 kB view hashes)

Uploaded May 19, 2023 CPython 3.8 manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (663.1 kB view hashes)

Uploaded May 19, 2023 CPython 3.8 manylinux: glibc 2.17+ ARM64

cutlery-0.0.5-cp38-cp38-macosx_11_0_arm64.whl (659.4 kB view hashes)

Uploaded May 19, 2023 CPython 3.8 macOS 11.0+ ARM64

cutlery-0.0.5-cp38-cp38-macosx_10_9_x86_64.whl (688.8 kB view hashes)

Uploaded May 19, 2023 CPython 3.8 macOS 10.9+ x86-64

cutlery-0.0.5-cp37-cp37m-win_amd64.whl (592.9 kB view hashes)

Uploaded May 19, 2023 CPython 3.7m Windows x86-64

cutlery-0.0.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (684.1 kB view hashes)

Uploaded May 19, 2023 CPython 3.7m manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (660.8 kB view hashes)

Uploaded May 19, 2023 CPython 3.7m manylinux: glibc 2.17+ ARM64

cutlery-0.0.5-cp37-cp37m-macosx_10_9_x86_64.whl (688.3 kB view hashes)

Uploaded May 19, 2023 CPython 3.7m macOS 10.9+ x86-64

cutlery-0.0.5-cp36-cp36m-win_amd64.whl (644.6 kB view hashes)

Uploaded May 19, 2023 CPython 3.6m Windows x86-64

cutlery-0.0.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (684.2 kB view hashes)

Uploaded May 19, 2023 CPython 3.6m manylinux: glibc 2.17+ x86-64

cutlery-0.0.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (660.8 kB view hashes)

Uploaded May 19, 2023 CPython 3.6m manylinux: glibc 2.17+ ARM64

Hashes for cutlery-0.0.5.tar.gz

Hashes for cutlery-0.0.5.tar.gz
Algorithm	Hash digest
SHA256	`77311d871d36f6b02137df584d3d5972ad471c1359649ae02980a2a92589afc3`
MD5	`72034f01c05329f434139f7ae75fe4a1`
BLAKE2b-256	`ac9e010b8994715568e7ceebaba00b914bcdccd6d4a3c16b7092ae80a0d82301`

Hashes for cutlery-0.0.5-cp311-cp311-win_amd64.whl

Hashes for cutlery-0.0.5-cp311-cp311-win_amd64.whl
Algorithm	Hash digest
SHA256	`e1df090b0c0dd10b57125732513c57096c1be4e38a896df43346b6fe0e1475e4`
MD5	`c7da053942e5cbbe4e99085c537212e6`
BLAKE2b-256	`4781f65fcec2b9d478ab1cec18ab3a2b469c85b756dcd10d556d2bb71aca9ffb`

Hashes for cutlery-0.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Hashes for cutlery-0.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`1cf6508e7734beae1eec6e88e0d0d78d00ffa95a3269d35517c4c285e0037c9e`
MD5	`681cca26473a8b504286a24c3acc5155`
BLAKE2b-256	`112e62e0cda878d7ab8058ead32e251b4030f8a6b5afc0c10d36cda813cb93f6`

Hashes for cutlery-0.0.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Hashes for cutlery-0.0.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`5586ffcf6c52042ac08ba98e7d47670c6ffff62cb987ad896c50994b4c4d10ec`
MD5	`0c83f6b3ba2ec8d4d79a0901fac23504`
BLAKE2b-256	`354f8b55b238169d6b087416b4f5154583945c4435347f3b2231194b85f06758`

Hashes for cutlery-0.0.5-cp311-cp311-macosx_11_0_arm64.whl

Hashes for cutlery-0.0.5-cp311-cp311-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`448ab9ac2d9fdb4f8b6abcd128f5352e1af61bc02a4dfccc3882136aad0ca57c`
MD5	`f0a907dc92ade1fc60361473ead962a2`
BLAKE2b-256	`25a80b69e13b310af1643a951181f9863613a2e4a8d9f4db1df853c8423ce676`

Hashes for cutlery-0.0.5-cp311-cp311-macosx_10_9_x86_64.whl

Hashes for cutlery-0.0.5-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm	Hash digest
SHA256	`8994bc96eec718cad8271aadd5567c3abe21bfaeedee8b819eda098429f01442`
MD5	`d43f40aed144d9c61f94a9e85e9bb2dd`
BLAKE2b-256	`025876e801b443fe1e7f893c0d7a298e0d55f885868df8c5a5baafe532dc8060`

Hashes for cutlery-0.0.5-cp310-cp310-win_amd64.whl

Hashes for cutlery-0.0.5-cp310-cp310-win_amd64.whl
Algorithm	Hash digest
SHA256	`07566553d107c89e5aca819c025e755bbbea81332566354ae9f39eb3a5f903dd`
MD5	`34f2b8538594b97e68cbc48df1f9c6ff`
BLAKE2b-256	`46921e13225def6e565c79918201265bb2bad939f2de219ef637da49ae11bed2`

Hashes for cutlery-0.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Hashes for cutlery-0.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`5ae0e599426eca2314b9fb5a4e9dbdba3e5b0173a8f0b0a02bc60b3970b3cc22`
MD5	`1af05e1e09cba07474fc5cad2bb1bb62`
BLAKE2b-256	`cbf8a3e379c990600cd6d3fe6926cdeceef4ab0b868640b8b2778624e9de11fa`

Hashes for cutlery-0.0.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Hashes for cutlery-0.0.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`96603bd95120225e5916825b0e6410065f19603c61c635627772c64ad32b0846`
MD5	`aa9907e2ad5b53d6ce6e631ff33a1f0c`
BLAKE2b-256	`3d4ce01bd2eab540c9d47c07e51957380f654b52d8111bffb9ced1a2d980db4e`

Hashes for cutlery-0.0.5-cp310-cp310-macosx_11_0_arm64.whl

Hashes for cutlery-0.0.5-cp310-cp310-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`fd4388fe27da6c0246c73b03e9d317345731576cf47dce917b904f750e0c3671`
MD5	`b794faa82680d261b441ab55af5c48e5`
BLAKE2b-256	`fc5dcfeb6d26f374e04d9f01c52f29fee330938f265783f225b750cb1bcf70ac`

Hashes for cutlery-0.0.5-cp310-cp310-macosx_10_9_x86_64.whl

Hashes for cutlery-0.0.5-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm	Hash digest
SHA256	`fb708476979ef4e49126410932290b83684c42244b76d92e6e700565f51245ad`
MD5	`5bed3443691cc20fe4573416596a9b13`
BLAKE2b-256	`93607cd16a7d0a3cf6ac20fe8d6f3be9735e83307f2da6f5b304010039633d25`

Hashes for cutlery-0.0.5-cp39-cp39-win_amd64.whl

Hashes for cutlery-0.0.5-cp39-cp39-win_amd64.whl
Algorithm	Hash digest
SHA256	`a95123d664b8fc46cd4f2ef8bf834d4c529478a9bcf42bad3f972c468826fab9`
MD5	`d8097bf8a122b244a299ac2b2f2a0730`
BLAKE2b-256	`fae5f5451b26e33df60652c8f48ad863203836350deb407dad39daf69ffe1b12`

Hashes for cutlery-0.0.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Hashes for cutlery-0.0.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`31dc1898578ace7ddf832a1992c9fc2feb469e6cd26f6e76d6319c416dd48d48`
MD5	`c3bab5f773b673e84f7b23531361826e`
BLAKE2b-256	`e75e184715f17efcbb9e7648891e0e5066f2e317d79a7e62bd14a4fb80093854`

Hashes for cutlery-0.0.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Hashes for cutlery-0.0.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`92f4c3009f4593cc7ecb6d7bc6a3a387198c8b4b2ef281572fe70dae09f9aa55`
MD5	`ad132de1ff16b29376a81f622fbba598`
BLAKE2b-256	`afbd28dd55bf346780b0791104eef1ec0059ef1c1792c126e514a8585a193d2b`

Hashes for cutlery-0.0.5-cp39-cp39-macosx_11_0_arm64.whl

Hashes for cutlery-0.0.5-cp39-cp39-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`787452c08ff95179809054c3cf8bf7d18b4eb0d1ef5260b3231c0db737b61139`
MD5	`b78e7f5af34b0b1127c00186beb28b7c`
BLAKE2b-256	`9ff127a9174c844d55171dc869986babd5d5f088dc3c17ee16cfa39036c6dd5e`

Hashes for cutlery-0.0.5-cp39-cp39-macosx_10_9_x86_64.whl

Hashes for cutlery-0.0.5-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm	Hash digest
SHA256	`a53495b681d9afc60410209e7bb2a8179e1279bad405fc6d265412dcf9319280`
MD5	`0a192ec4621e49fd0fb82891c4748e53`
BLAKE2b-256	`a3cf504f8c7ac78706069b52c62c4ce6e7311f143810af831291cb6f74618f20`

Hashes for cutlery-0.0.5-cp38-cp38-win_amd64.whl

Hashes for cutlery-0.0.5-cp38-cp38-win_amd64.whl
Algorithm	Hash digest
SHA256	`118bfb0cdaefceec8190b7c385fde5bcf7573374d8991e960706468715567c8f`
MD5	`83f5da33949e2ff995e68fb021334b60`
BLAKE2b-256	`e459b47db3b5c725ae79a54c5015ee6647006264b2e9578096fd7ecc4c62138f`

Hashes for cutlery-0.0.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Hashes for cutlery-0.0.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`60f6b6d4fc6ec2f37fc13ac4550dc4a79b89233840a73d5b9e8ae50dbdc6e441`
MD5	`670e918bf30f55678a90d43263cbe24c`
BLAKE2b-256	`b0ac964f73fdbdf85e1e91b2b5baa802efdaa4a415a768426e4b9430f9c9a1bc`

Hashes for cutlery-0.0.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Hashes for cutlery-0.0.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`4df40fbf997e19cf9d3f9f3da1f26d23d63106001c2ef0e0e7286a7199181e4f`
MD5	`6e63c6267f42b069e5858e27d7cbbdf7`
BLAKE2b-256	`ab9de3be7e4e5ab545a02ea504778592253695d9e531c84c40da0edb0d6af46c`

Hashes for cutlery-0.0.5-cp38-cp38-macosx_11_0_arm64.whl

Hashes for cutlery-0.0.5-cp38-cp38-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`0ae646644857c18b8b1df276f6f7cda3860d99bfeccd57411656e578d65150f2`
MD5	`a6b4d9637e8a8832a787fae58333f975`
BLAKE2b-256	`6705f13dd358ed3165090bc6e184e95c1a2b90b12d7919df06b8b2aaef07acb2`

Hashes for cutlery-0.0.5-cp38-cp38-macosx_10_9_x86_64.whl

Hashes for cutlery-0.0.5-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm	Hash digest
SHA256	`11e6a067211d3a0515e18a257cc28e488677e990231dbbad4a71b4208be6fc19`
MD5	`fee5e0d40232e8b00f9f3deccb70ddd5`
BLAKE2b-256	`15d5b13af14595fc03bb5a20a2a4c8605d1be2388d17adb07ca7c3d7b4bc590d`

Hashes for cutlery-0.0.5-cp37-cp37m-win_amd64.whl

Hashes for cutlery-0.0.5-cp37-cp37m-win_amd64.whl
Algorithm	Hash digest
SHA256	`3199d8dd6dd804b938ab78b3befb755c3813cce73cf0c57cf0579b34f70c9ef4`
MD5	`524691c2afde2b65fe04da95d38faefa`
BLAKE2b-256	`c85363c5873898ef93ade8ca1e3ffa0985a79595a417bd3d5b14ba2116f26008`

Hashes for cutlery-0.0.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Hashes for cutlery-0.0.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`f6d9e2b6f52cc42cc52523fce13ef19ceffeb9a41de902d87c5cfeab1bf537c0`
MD5	`0a054946ed7cace258f93b25b8a66ec7`
BLAKE2b-256	`d1f35e1836e66df0c466e466a793cc95b3940ac29b91c7eea4833abb719d5fc1`

Hashes for cutlery-0.0.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Hashes for cutlery-0.0.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`5501d48d906c5a22668843e9c272ace7b9c5e39cd5f927d0d1c733fbaa54abe2`
MD5	`7812a490624e44278f33415a4ae7079f`
BLAKE2b-256	`7ebeec3c4843144f03168b10750ac37e95c72d6a65ec36a9f24936099f5a14fb`

Hashes for cutlery-0.0.5-cp37-cp37m-macosx_10_9_x86_64.whl

Hashes for cutlery-0.0.5-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm	Hash digest
SHA256	`b7e97f65af100ea7cb66ff48cb4c17f717e54e5b715ed992c8d4dbc711c2f2da`
MD5	`4a46b5e2469bc04ba29b4d781086eaaf`
BLAKE2b-256	`b67e07dae874c97e4028ee3d909a15a39a3d5f1e58a0b505b88103a2d65e417b`

Hashes for cutlery-0.0.5-cp36-cp36m-win_amd64.whl

Hashes for cutlery-0.0.5-cp36-cp36m-win_amd64.whl
Algorithm	Hash digest
SHA256	`72bf3da823c18654d6c3a59c8577601af3cea8007f3ce089ef591b536d34707c`
MD5	`ef99c2d99dfbf398caec8578f6c5f62b`
BLAKE2b-256	`8b449cef45be71c3643aa9c0017a310ce71a26a0840cc2f478ac697381c5acd2`

Hashes for cutlery-0.0.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Hashes for cutlery-0.0.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`387515287c0ac3cedfce4107b205759661ca2397a64b7610364900af37296309`
MD5	`cbc65c6426cfb6fe7f775e7fd44595fd`
BLAKE2b-256	`82195728df6a55355fef6f839a2e36d4063c3aff1e18a5b66b333657849beea0`

Hashes for cutlery-0.0.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Hashes for cutlery-0.0.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`87b47296cb8c0f18188d3bc8eabe1668e227c7da391aee725149ed4fa828dac1`
MD5	`3513e45ecf6fdd142c26679d0dd62182`
BLAKE2b-256	`26e10739fcc2e308ed1bd3baf91e235a2d7189640bb79ef0066e99823bbb2074`