No project description provided
Project description
🪙 toktokenizer
toktokenizer is a BPE tokenizer implemented in rust and exposed in python using pyo3 bindings.
import toktokenizer as tok
bpe = tok.BPETokenizer.from_pretrained("wikibpe.json")
assert bpe.decode(bpe.encode("rust is pretty fun 🦀"))
Install toktokenizer
from PyPI with the following
pip install toktokenizer
Note: if you want to build from source make sure rust is installed!
The only class toktokenizer
exposes is BPETokenizer
. The class itself is pretty minimalistic, with all major methods being showed below:
from toktokenizer import BPETokenizer
bpe = BPETokenizer()
# train a byte-pair tokenizer on some corpus
train_corpus = "this is some training data. any dumped string will do!"
vocab_size = 8
bpe.train(train_corpus, vocab_size)
# save tokenizer state
bpe.save_encoder("8word.json")
# load tokenizer from dumped file
bpe.load_encoder("8word.json")
# encode and decode
input_ids = bpe.encode("some data")
decoded = bpe.decode(input_ids)
Performance
slightly faster than openai & a lot quicker than 🤗!
Performance measured on 2.5MB from the wikitext test split using openai's tiktoken gpt2 tokenizer with tiktoken==0.6.0
and the implementation from 🤗 tokenizers at tokenizers==0.19.1
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
toktokenizer-0.2.0.tar.gz
(59.2 kB
view hashes)
Built Distributions
Close
Hashes for toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 069783d86b946fb1763fe56b2628dc8faf631d629c4a8b7682270e3ee10a7751 |
|
MD5 | 41d22ed86c6021954cc1f80383ddd0f8 |
|
BLAKE2b-256 | 044683deb1b4cc0c4edae6c1372e8c165c1d235bc39cb57b54be52bb0238db5a |
Close
Hashes for toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2424f5c30081e1062345153253a790e6e5b9fc81422b769da41815ddc195bf11 |
|
MD5 | 6683a5d6b931a4ad434afcd2e43a887a |
|
BLAKE2b-256 | dc07922e57445b35cb10bda836e84262bc2a296b07aa47c6fcc69f8b49597b14 |
Close
Hashes for toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0a38a0fa267a6ac55de5114bf81f1bc4b250e847897eb1d59e53884e457a8f8b |
|
MD5 | 657828df1744585d73535a5c57f05587 |
|
BLAKE2b-256 | 50dbe41617a872c1873b13dcdb3df0da3511629d2721d50c101b66d441799394 |
Close
Hashes for toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ebb778bbddba7bebf53b0d5e6b53368257cb8130ae9911a92d98c8391f7766ea |
|
MD5 | 0a3be755da270ee6b682fa458c09602e |
|
BLAKE2b-256 | ea68382167126121376a24ba4beec48aa478b5e1ba5d80c30539e51aa67f0676 |
Close
Hashes for toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f57a464907d46a07edc132b98ce09131e52bfa801df91ffd11a43e276b61c3d |
|
MD5 | afbfbc2430291a5bb3fdad765c9b8c8b |
|
BLAKE2b-256 | b837699ef680585e418fb1e42092fe5b2e20e7096ef36527d6874f40e019ad28 |
Close
Hashes for toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7f463a98f4343eba3b4ce39b307b6d10fab8e8d85eb54c5c959ba7993f375f68 |
|
MD5 | 9f47dda401c0e4c9d89e6a446077253d |
|
BLAKE2b-256 | 718d56bbe7d8b5cee411b255ca8348f54d6b01af8e0bcdf9ff4de1377f7bb418 |
Close
Hashes for toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0b4f5ad04f48bd83fc8da7b8a26022248c4e8de0d9152b7d3ff64068498ff47b |
|
MD5 | c6f07e7200b286f1bc3e758a1b22e714 |
|
BLAKE2b-256 | 47394b00a1ee2a73ebf9a7898f7eff0ff1fa2ae4d39c64865c1efd20c8eaf6ad |
Close
Hashes for toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 817f532d3fabde2afd481b8cbd1fbf7dfa5cad5b1588b6d43e12df96c3d27a7f |
|
MD5 | a6e17eb39090b2f3ea719bc01354ee28 |
|
BLAKE2b-256 | c1b2d98fcd53e0952a82bd9653810a16ef160d97c52542a352237d77526b302d |
Close
Hashes for toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8c1975c0f7f8c36ae9fbd801a12f35e6afef2d0bb1207b1bcafe2656322e72e3 |
|
MD5 | 9b7d2af3bfcd9baa4ff6cf4e5caa67e1 |
|
BLAKE2b-256 | be732dcd8482d012b2583b283a0504a4dd61c02b4b245565ed6c1b84b386656a |
Close
Hashes for toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d11fa5820cd071eef913ec6aadcc00d85d88d5ff146e408f3ad2bb82628091d7 |
|
MD5 | 36e201f44522855a8d6021c9c93ff37c |
|
BLAKE2b-256 | af8a7da83b0e1e3d1c0fef03d18e1fbb59d7e990320795728f65bd77773424a6 |
Close
Hashes for toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93a4ef6e3a09c78636149f216ab40f360f9408a92d8d83a058200142e9d66fb3 |
|
MD5 | 854150990792dfcaa9c4d2a545ce4b60 |
|
BLAKE2b-256 | 432f345678dd2af2d60573d674a971619aad60667ade0e19940d59d01339bbc3 |
Close
Hashes for toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d4c1d19848b4c61be7fcd8719e1e7309c9f0a445775b668cea8c571d18894e1b |
|
MD5 | a0931a0761602562c9727636d5a9bd84 |
|
BLAKE2b-256 | 45dd3c44e13e80fb490026161b285651c73e509e57ce401f6aa3d48b423b311b |
Close
Hashes for toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d07644b4fd4d98024b6c65fe2378d95977548b35e8c3416cf19d45aeceb82bb |
|
MD5 | 655e519b8acacab94721d333b3da0069 |
|
BLAKE2b-256 | b1c1e36c02a4ae8dd03307fafa617b262c2080147e1f55484eccc6c76fabef78 |
Close
Hashes for toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aba963b4c1f3ae074f29ab1173fc2a70db81924e03201f1556d7fedb917b43c9 |
|
MD5 | 9e53aeda9a4d3a188d7d9350ab2083eb |
|
BLAKE2b-256 | df145a89926f3be99f3894f37344d6f61ab5c75c9b60d41bdee9e9a3cb081f27 |
Close
Hashes for toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa703437dd77ac85d375fd21f62e4ece3ac2ce23d867b4f6613f7f87ba215535 |
|
MD5 | 82886c5d57008e5c1a0e6570a7efac9d |
|
BLAKE2b-256 | 4fc3d5e3473d6e40e288377fb5502089d59004a5e19b0a7cc90f3eb9a853bfd2 |
Close
Hashes for toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68689fee266fdfe5dc145a376f6719c019d6da59586a9ae975d1605438f55b5d |
|
MD5 | c2ee906adcfdfe84b058335e36c80054 |
|
BLAKE2b-256 | 02bdb02cf63e8d461e5470158f23b309b95b86af3547496e9dbdf18030835423 |
Close
Hashes for toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c4878f280f8a869d9a9f24240b9b4406e469e8cd250d974396ca2ef936eaf05a |
|
MD5 | 184c619add00f2593512f0b656458d5b |
|
BLAKE2b-256 | b68ea0fbeb326dd9a367d9e814f0ffbbdf1cc9b4091d79a166c5183d9e074bf0 |
Close
Hashes for toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1662d2a1ae9e7926ae58d0c5da3e7d7593f30a0b04be7adf979c12d8441432f3 |
|
MD5 | 77c6f42567b3f9761bd044ece6db5410 |
|
BLAKE2b-256 | 4c2b20b4a5cf50f42ba9e19308b995108462470cb0155263474930651d3f165f |
Close
Hashes for toktokenizer-0.2.0-cp312-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c875c8ca025853677f00f1b3e38245b99b081514746a34d2fc33d64b3ccb4c95 |
|
MD5 | bc9da1e9f7cf1cbedb02779480bd0e6d |
|
BLAKE2b-256 | e3f33f9fbcdb671e3af7dd9907a30ee6480cd1a06df4b39fa6268a221086821c |
Close
Hashes for toktokenizer-0.2.0-cp312-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b1b463046dadcfecaa27a3efb609f16023d31f5324379b59c4e79099336b209 |
|
MD5 | b48af3be96f216e8868150ed4f41d05e |
|
BLAKE2b-256 | 5488346daeeed2c43aca09b60b7fe61e5b14b54c5185a483d28289f00250ba0c |
Close
Hashes for toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 60637c4506f02109b0709cefbf0e38a2c67abcde0856961cfb5836a9ec999ed0 |
|
MD5 | fec3dcf5218fb158764c91c791bfb37f |
|
BLAKE2b-256 | 4c8970f5abd24665dd5bf14b07ca6d5ab5f0b429ae3c03093bcaa9ef92f35c9b |
Close
Hashes for toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddc8d7a78e08dfcd4ad6e9b535c2de11f8a1de21b9c028f898466d39098f473c |
|
MD5 | 4f6d64e73b4a9cc725301d118706e02a |
|
BLAKE2b-256 | ca8ab74ffa9fac4397c5e0a4cd2333c45a47b401a050388ba358cf69f42ac69b |
Close
Hashes for toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d0e657d0bc6e42a23295fbf8752e584c389d45ee97aa74e188cebbf49bea87da |
|
MD5 | a64f2b3d36f2bb758faa30750f3bc0c6 |
|
BLAKE2b-256 | 2ca7438d96ab63bf42a25ed9beef90c59d41dbb544789cd51cf98f902ab082a5 |
Close
Hashes for toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eca1eb616896ee114351b0c84f17a82ab6fd36addabe21242a4528045849610b |
|
MD5 | 2cf00919861728db6ccd5b3cdfb2fbbb |
|
BLAKE2b-256 | a6536d3738bd73891d38df78523338c6b8b3c78463787858ec0d11e198da21eb |
Close
Hashes for toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b30e5309b6740f792d97b3a9c81f367823cf5a2d108c968344d459e01426dc1a |
|
MD5 | db12baf482cb5c6f0a52cecc07d6a41b |
|
BLAKE2b-256 | 2d95aada8ea079b24c3cdb35dd55be6dc3a5ab3927f171300b0e38f33252cc13 |
Close
Hashes for toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bee6cf40fd431d9d8941af90479c67656ab79e4a978355777090bcf0e4303a21 |
|
MD5 | c7051a0fcb466dc0dbcd75e74a679329 |
|
BLAKE2b-256 | a8bb864250780722099245bbf3a3a83f23d9cb102db7663dad06ea09ac69837c |
Close
Hashes for toktokenizer-0.2.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a77fbf1912d0b47fd894257db15b17168eed824367ab0e2160942a6aa367781 |
|
MD5 | 660d6609a4ff371e1cdb5b31967f4f18 |
|
BLAKE2b-256 | b5e240234146d28937dc6ed280a58516b321241ee1334dbea1f537db2f6a10d0 |
Close
Hashes for toktokenizer-0.2.0-cp312-cp312-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd42f9ad8e56380985256857321c7f85fca4d19e3b6f703c1e7dd7b44a5606fd |
|
MD5 | d51c70e8e728b0f14cb61611f01c33ab |
|
BLAKE2b-256 | fa4a7beee22a1d01e22f0c127d5a49bf458c59984dd64307924d98a18b1255fc |
Close
Hashes for toktokenizer-0.2.0-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b9f9cc122d0095791b1a9fa2deb1d3186e2cc6e73d3d13ea8d017451a3df252b |
|
MD5 | d6e5b27a6e4e25caad0af51e807b3985 |
|
BLAKE2b-256 | e6764c5e0c3a204236054a569ba3c161bf9df27895416d4aa05e927384626f29 |
Close
Hashes for toktokenizer-0.2.0-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f12b4c5f3d8ad53ea6a0bbf9894906143acba050ad9237113e3680041818c9f2 |
|
MD5 | e7fa433c8e237d3b76ea46bc10e227df |
|
BLAKE2b-256 | c3432ebd0a7626fbaa1b91a92d0a5ebe23eb587967cc61496e889dbafc17b8f1 |
Close
Hashes for toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e2c6b224eff0c16a10af33fbe93df794e20671c628eeed514814b6e84f3e8067 |
|
MD5 | 79e6d65c17fac116bf1a285ecb288c1d |
|
BLAKE2b-256 | c19583337c92b42d91d96f9e7fe770882d838c9d3011d2238bafa93ac72d01a0 |
Close
Hashes for toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 195b9f551e99a698077e5cf8bef926df86982b3caee980afa748e422262877dd |
|
MD5 | a6d9c4b56b08f191e78261f850e1cd82 |
|
BLAKE2b-256 | 96e79afb592efd56c45030a867503fa5fb2989087c2d9de0b693f9d5ea74a8a7 |
Close
Hashes for toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25b7dfd9b2d1ba4b127e464297530d83a9afb3c17ae482a55d9f67a50b75ff39 |
|
MD5 | 31dd1a3135095d304d0e63e7dbf41e22 |
|
BLAKE2b-256 | b76f7d9981f854c6f7ff8e4c11a069e2b6fc79aae878ba029c07259a79175a31 |
Close
Hashes for toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8ff839dca9d04b94bede3d8270ec02c739f0366ac7847c1fdb467ce07041a6b3 |
|
MD5 | 03598d9596dbe1aea9719f5de007bef9 |
|
BLAKE2b-256 | 3f5f506ed6714539b2caaab049290b93c95d51ca8bf6bf3fecb5aa32a114e806 |
Close
Hashes for toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c8a2c43047f207fcfc8341e1433f85924726678c1ce98435df35a32ae3f9404 |
|
MD5 | fe47ae4a3e896ff9d8fe781eeb486683 |
|
BLAKE2b-256 | 33087e2ea4b528b812e0904e758454f0d3c555471e7a88a71def04fd2a1d1aab |
Close
Hashes for toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f0afcd7c6856e5550b3a9eb7ea20323e1b6e8dc6d08444ab5c77b244567da294 |
|
MD5 | 7ba620e74c872518ddbdebe6c6e54743 |
|
BLAKE2b-256 | b2073325b4b58112a2baf8fa9609dc75e09428b4bdb4ca0f2564b472ac9fd150 |
Close
Hashes for toktokenizer-0.2.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04ed43770a4517ab32e8fda0f371b844136e439c3bce51cd49a743a5434674a3 |
|
MD5 | c1067b94df2312495763c86f97f6aba9 |
|
BLAKE2b-256 | a4092a7d0f0872a8d4d0a3e65b7bb0667340ec006442c30494e897e3ccf0cc5e |
Close
Hashes for toktokenizer-0.2.0-cp311-cp311-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 50340c409021c2a59cc6cbee2f03728c0f5b0bb5779f6546e46a76fa5dbfcedf |
|
MD5 | 38ce3afb0d1dabc8e677e06456f49a62 |
|
BLAKE2b-256 | 98d960c6f3fa1d96ab630a2a4ea2939ee4bf8879f085b2e743ddb194f502947c |
Close
Hashes for toktokenizer-0.2.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47282907284791c4a3b80b74b565e48bd2c8cadd43b3d06019f85e8f2f9a0644 |
|
MD5 | dfa0330953815a9dde73a175b631fd19 |
|
BLAKE2b-256 | 6e637353a5189bd834775b13fbf03ddeafa39bab7ded6ec82a51de62be304f72 |
Close
Hashes for toktokenizer-0.2.0-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0321ca0530c941bd322e1d08225db0973d1165d63d94b184da7d87cfc35cbf9c |
|
MD5 | eadb105365d021fcae8eef8768d6f85d |
|
BLAKE2b-256 | b779fb361eb81809d70db0525eab98d66ab2a56c6bc26a4766f58339897c861d |
Close
Hashes for toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f9c6b45ef770b74908cc6f3c18a787ab9c1ad7534ed93fc064d5e09954a1b62 |
|
MD5 | 75e384ae35289c9c6e87280f0c1f9334 |
|
BLAKE2b-256 | 373d9ec9c43a10e2aa8cd891f76c5f428f46fc38b1bc12f0abcd6a6628d3fc4a |
Close
Hashes for toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ac5944749f2bc357d28db630e185a33f6f07f50d8f2c34a450122f0bc4bb4f7 |
|
MD5 | 109fc476c36c2d9e18a6c5073fb24e9d |
|
BLAKE2b-256 | 2c3d93cbdecf985ed4e97c76d3bce90ba8c3ad8f2a0b5a23808305e028f63cb5 |
Close
Hashes for toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7bf30ea08bb5949b54b21047b06726f9a3b90aedbcd4f765c01d5c157f0805c |
|
MD5 | f00f713bf4c985eea0834b62699e2b7d |
|
BLAKE2b-256 | adb365d6c9e5fb806e37f52260a1055a4113d807d74e4de1b4999e876b66e554 |
Close
Hashes for toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbe3909377ffbc6b44bf25635545e87810d94bd3266024771ad4e22d80f96716 |
|
MD5 | f872c8f6502b4193d1226cbc46df296c |
|
BLAKE2b-256 | b60bf89c1ca1604f02ccf121b6e01c01b48b6c695e7f0ef50d348bb751ffa232 |
Close
Hashes for toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a46e71baa10fee438959cedd5ec54d4b192b56ace2051a7a45ba63cf982d13d |
|
MD5 | 7791b9fdef8f378081ca2a8afb3e0261 |
|
BLAKE2b-256 | b331bbf22dd34167c979e7ff057964c3789f32fdf420bfedaa6e9d5ff8cfbec7 |
Close
Hashes for toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 06b146616ffb061bf3bba397b39e67adca08cc160ee625c7bdc1237cce81b0ee |
|
MD5 | d612c7603b68ccc67dc299c4075f5fe5 |
|
BLAKE2b-256 | fceec6991c32b98665850c2221249a97dc403daf561d59e329f1d2c80d97e30e |
Close
Hashes for toktokenizer-0.2.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f726dc9a4bf9fe3973e21db4237bd77f6e419e703d9dd8316d6b0c44107cf977 |
|
MD5 | cb75493fe9b2e84ce7288dac6d0f9925 |
|
BLAKE2b-256 | d79f9f00fc8cfebef654733a48ec28bf81aaac2485800de97280599782228976 |
Close
Hashes for toktokenizer-0.2.0-cp310-cp310-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da93360e69d292db64243b3939164e094c0894e651a139b60644d1619542b90a |
|
MD5 | 60aff3a54153d75c33baa5fa129cf06e |
|
BLAKE2b-256 | de47b2afbc5cc9c6bd6211f55339679a14d1674ff338c084cba44af84643a4c4 |
Close
Hashes for toktokenizer-0.2.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7cf8ac3b4cd26f71aba46f92b64f643ec757a47d3a7ed61557ac238c5bdfb6ca |
|
MD5 | b4850267a81f076d59096de732a0d97b |
|
BLAKE2b-256 | b74815b8f044ac2e041f924b116cc23c99023354b97f5b17370ab365d3c823b7 |
Close
Hashes for toktokenizer-0.2.0-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07fbaada27280274428b138b16f936c83c390d523e749da7cd6943e7db14a4d9 |
|
MD5 | e3ac07ac6b853521fe68a3cb78de7fc0 |
|
BLAKE2b-256 | a08c427447c97a246e4b701356474e009fb8c27a16141d00f7e0dd199e4f7ea5 |
Close
Hashes for toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca0845b6f0a761999ffcd63a788f60aded429a91e5bb85ccbdb0d1b468a30f8c |
|
MD5 | e8d731d6cb3a2ef70ecfcb3b5401ad8b |
|
BLAKE2b-256 | 1967aa0637c86264939eb2ca96213f77a495eeea5309759b997e84dce7a510fc |
Close
Hashes for toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9dc4525fdf7c7f7a4f1bafad125e898625897a5578649c37a3020872e79afa83 |
|
MD5 | 6990301ce262e89f1d0d08efc0509937 |
|
BLAKE2b-256 | 09519817b8f6322cae509e865ad3f4aef6134ca6bbfe3b594bc0e347e26e93fa |
Close
Hashes for toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a6d147b943528c4117a96cc924460014532d9c4a332361b9d403b38849c3d61 |
|
MD5 | cc744ede92aded529d158486ff2b157a |
|
BLAKE2b-256 | f04be42226ee7d9af6492356f16f435c5bb25eddc91662ab32a0ba3bc2fe5dea |
Close
Hashes for toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c1bb30cd2117551dc46b1c5c00e78ec11e80f9549218941c9dfee4f16596dd93 |
|
MD5 | deb0b9f4bff926d7c0b40fac8418779d |
|
BLAKE2b-256 | 78836e56c09df18bf7c7080cc7d72f03719b2868ce9134664a5b16f04eb12206 |
Close
Hashes for toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1acc2b168fe6d1489a54bbf91fec99dd8c0b2d6dd11aa69562eaf1c50fdf2f6e |
|
MD5 | 72570838931da758a87d5c230a707139 |
|
BLAKE2b-256 | 6813f0024b5b2851b260566706faec4534783bc6f881359e67429929366822dc |
Close
Hashes for toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84be565413e2722ff4f45c456656989e8df8a4f88313abdace89cc14a3d5dc43 |
|
MD5 | 51cd11b193a20ade814f077fb149cbb7 |
|
BLAKE2b-256 | 93e73549c9f56455af222b339f046137e08b5b2b5d0807b10def8ad8c8da46b6 |
Close
Hashes for toktokenizer-0.2.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | baecec8abb2fe7b42738c4d4943310d931622243b97894c181bed91cf7afe841 |
|
MD5 | 4f32d1dce5966ecc85e878b04cfaa331 |
|
BLAKE2b-256 | d7906ad98be2ac8e372da125077ba73c5b44753bb8c47f4460e89ec7b986fd70 |
Close
Hashes for toktokenizer-0.2.0-cp39-cp39-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 06ed09a9d618b5d95ffb2c54248497ff4d235fa0c698789bdeb8b1b723442a20 |
|
MD5 | 703bdae2bed728a5e4c1040e53b36863 |
|
BLAKE2b-256 | 68d05fc00d1a40be990541ba127d29c56a5c5d6866d94f9dba594fe35efcfc5d |
Close
Hashes for toktokenizer-0.2.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d08a5942f9443cd9346ddec7802abd5fdc17d7421aca3dcd89a33b7815822bd1 |
|
MD5 | e1f6d444a1f7f6cedbae17739375f8f3 |
|
BLAKE2b-256 | 59187c1fd5b377ebe833e9bdc644dd000a92fb75adede8435de16e5a259aeac2 |
Close
Hashes for toktokenizer-0.2.0-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13bb7b3b96ffaaae6bd2abd59f38677e1c82470dd977ad3252f4f45863a5b43c |
|
MD5 | 1378e81f6d48853b326928a92cd55c07 |
|
BLAKE2b-256 | b290bdaccca83e054c973fcef97aff48d4131ce9ab5cae6a48266edbbd6ff075 |
Close
Hashes for toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec8c71a4a20d5411b9e51da0a86c960d901e210c0b64403fc64d164c72f6ef8f |
|
MD5 | 93b6338987dfc57c2566e8ae999f9ed5 |
|
BLAKE2b-256 | d2a2d5231ded9516c43ac499c9331cba0c309944f84909e4b3497083e95496a8 |
Close
Hashes for toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 533d79658b39436075203beeac50bea4f50e76ac21dbbf22780ef36a152ed59c |
|
MD5 | 1a24d56b26b08cc65c28e3a8d4568baa |
|
BLAKE2b-256 | e648a8754785ca21ceb54c8502cdc50585362f9711048b7daad4c2858f055df0 |
Close
Hashes for toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8669b1a6dc055281a803e0c8925f4b01f19ef625521a8cbccb4f67eb34e9c822 |
|
MD5 | fdd47063059210029788d51ad29b5d99 |
|
BLAKE2b-256 | 448456e0efc0dca43a9bc7c95ee6c985202654cae4be43993323e218b4eab00f |
Close
Hashes for toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e488021c1b9a1a573f87f3e22708950e5617fb8f6ba37657f4c837e4d7ba7af |
|
MD5 | fa3eb3bce865324225dd8eaf92558a9e |
|
BLAKE2b-256 | d497bf9759ac0c74ac5da0d698153f55d56b074724c53685149f1915b9079e2a |
Close
Hashes for toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d96e9e8c75b78686575a8f8f2f9bb8739f4c2c0cf5789b3f15a636c55a6f53ab |
|
MD5 | e211a981742456ff3a9fb250f4b9ac63 |
|
BLAKE2b-256 | 57132eb18a4c49b14a7e22bca24c6404616cd0b68cb7431d953ad4ff80cdaf16 |
Close
Hashes for toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 609303991c8d2da9b6082927ffd97289093c42880a08a3e040efa9d919084148 |
|
MD5 | ba6b534f25646face74debdbffbacf95 |
|
BLAKE2b-256 | a3ed002a58b3f4c555f3fc415aaca2cff46e5a6d4c6fe03fefe054bdd971ff1e |