No project description provided
Project description
🪙 toktokenizer
toktokenizer is a BPE tokenizer implemented in rust and exposed in python using pyo3 bindings.
import toktokenizer as tok
bpe = tok.BPETokenizer.from_pretrained("wikibpe.json")
assert bpe.decode(bpe.encode("rust is pretty fun 🦀"))
Install toktokenizer
from PyPI with the following
pip install toktokenizer
Note: if you want to build from source make sure rust is installed!
The only class toktokenizer
exposes is BPETokenizer
. The class itself is pretty minimalistic, with all major methods being showed below:
from toktokenizer import BPETokenizer
bpe = BPETokenizer()
# train a byte-pair tokenizer on some corpus
train_corpus = "this is some training data. any dumped string will do!"
vocab_size = 8
bpe.train(train_corpus, vocab_size)
# save tokenizer state
bpe.save_encoder("8word.json")
# load tokenizer from dumped file
bpe.load_encoder("8word.json")
# encode and decode
input_ids = bpe.encode("some data")
decoded = bpe.decode(input_ids)
Performance
slightly faster than openai & a lot quicker than 🤗!
Performance measured on 2.5MB from the wikitext test split using openai's tiktoken gpt2 tokenizer with tiktoken==0.6.0
and the implementation from 🤗 tokenizers at tokenizers==0.19.1
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
toktokenizer-0.1.2.tar.gz
(649.4 kB
view hashes)
Built Distributions
Close
Hashes for toktokenizer-0.1.2-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b11fe951ba6084608b83c1b8e0a9280132e20393f916398fea6f6bac158d3542 |
|
MD5 | df40228278d2f8cc904ed5fce4428fed |
|
BLAKE2b-256 | adec0f92960d823a1f7dbaaab7f23d6aaad0ad824d9cd8034b6ded1d39c5daa6 |
Close
Hashes for toktokenizer-0.1.2-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68f000876aea64ba957723c0b0eaed376004d3ffa9cd12bc77c6942bb85f15ad |
|
MD5 | e60dfea66db8e4dcae6e81823fe7a0bf |
|
BLAKE2b-256 | 868c8e967cb2da35a92cb3feb992f3539afab1b3aac35bab47e8ac6c9562fabc |
Close
Hashes for toktokenizer-0.1.2-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e7b5aeeb89f76414e9e695688769fb83bfafdb37b29a72588997485267a44921 |
|
MD5 | 00b47f6f88534d9541491ac68a627d95 |
|
BLAKE2b-256 | df2cdc4309a22bdbc5507b51a56447f01e5f794b6a77498341286937571efaf2 |
Close
Hashes for toktokenizer-0.1.2-pp310-pypy310_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 635ac0c234bff231ef103fd1f54971a3e271be58cf7a70c69f1009dbfde2efec |
|
MD5 | 28bff980e981d5fe08bcdeafa1fc694c |
|
BLAKE2b-256 | c65de0ee4a9a4d33bed7707f5d88c87d7301f8f547da322daa25a5f082bb0648 |
Close
Hashes for toktokenizer-0.1.2-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7078b67a4c08d7ab661162a7e26bfccd5af5e8b2a2479b2cf6688cff08de2205 |
|
MD5 | 81b6217fc35a47983dffb6aa4087bd37 |
|
BLAKE2b-256 | 98e7655a17545ed73491fa1c9a1f30b2b41d88d5a5d3eda69a855f03e9b3c888 |
Close
Hashes for toktokenizer-0.1.2-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9fcb84562464502458602bf196ec7259219adcbec9e024e35384d739d431c8f8 |
|
MD5 | 9e82f54b5618a489271d80385d804edc |
|
BLAKE2b-256 | 6523598012acfbea2a6893cd1361bad12d18e585cd208254e0976d0f087d5212 |
Close
Hashes for toktokenizer-0.1.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e03d82ca14aaaa3581505c1f4d56057f1666d6b1ce0cfb80b557d77efc5f1540 |
|
MD5 | bb5869b46b2c02b92126548dc1a1ffe3 |
|
BLAKE2b-256 | b522c20cd59f6b3d2f28bad87252e441702c7d3bfc396cfd9ce4fc2c8600c36c |
Close
Hashes for toktokenizer-0.1.2-pp39-pypy39_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d5eae019402a88d2c4bf37a52399e568fdb228a02b3acbff06b42eeaa697e021 |
|
MD5 | 5feaaacc7f2a776c3c364d396bdcc8c4 |
|
BLAKE2b-256 | e483ff2025d84d02ae4cde1f7b9d5666a12761453dad658b3d0f9aa34c2e8b8c |
Close
Hashes for toktokenizer-0.1.2-pp39-pypy39_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f01879c4dd5fc9d46bedde314f4eaab5dc8cf8e3ff189a71af22a475be6c0f18 |
|
MD5 | 7584d47b995b018cdffa3134a5fa54a8 |
|
BLAKE2b-256 | 24e118caa2b79023756026cec74a212ca3031df57b8719b551c540805a4b5267 |
Close
Hashes for toktokenizer-0.1.2-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 70a6f03b08cd41fbcf5559efdd2bd7b4306c07fef598a3d6ecf6b521328eac5b |
|
MD5 | e9aced3c309df3824cead4a4e7055220 |
|
BLAKE2b-256 | 93d4cd2b03bf9bd5312daa070a6be09191a23e1df3caf098d6db893217118a67 |
Close
Hashes for toktokenizer-0.1.2-pp39-pypy39_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f803789f6d6d551620ac08a6c39bae06ed9829e386bc83860c094057b22fdf6a |
|
MD5 | d417532e472c01cd6c9c0341bbd99bb3 |
|
BLAKE2b-256 | 0d2879183b437bca864de1a0f811657e00edce243be5a14f2ef93f705631ffdd |
Close
Hashes for toktokenizer-0.1.2-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f821d6a858aed6b23dfe8663cc05c16df2c3cf7568170ee56eb898bdfbfee6b4 |
|
MD5 | e302a26c051c0374ab58262f53fc8d95 |
|
BLAKE2b-256 | 6c840e29a662f8ae245cdabb0bec3c7384858109be0f56785fd2c34f931b831e |
Close
Hashes for toktokenizer-0.1.2-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3cdc1a2cc33a476864c5545de420963a71a803bbaf724153b85aafbcf298c137 |
|
MD5 | 6e20c82fba9ee298c0bad309aa87e9b3 |
|
BLAKE2b-256 | 66b9725d496832a543574ef64f5ee6e85fc3cfa5677c43d2b5e85cb84fd36291 |
Close
Hashes for toktokenizer-0.1.2-pp38-pypy38_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9eabd58d99602136ca535e6b78c6af7d414f26c24762de205c3a76241f32c6cf |
|
MD5 | 271f5c002e7bf78db201f079277bb547 |
|
BLAKE2b-256 | 961d4d0afb533adeb5be28cf607b041e3051e7676de942256b7fcbc686787bea |
Close
Hashes for toktokenizer-0.1.2-pp38-pypy38_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c680dded9b5dc8b0db89a3dc5fbab4700d641db313b861a741dc9b052c8dcc60 |
|
MD5 | affe4db6f3c0543a5a21b11927f8875f |
|
BLAKE2b-256 | fee8d8fbfb9126a2e4d4902c75b41b7b17dbb73b7d0d6fbc150b8a296d1e8bb3 |
Close
Hashes for toktokenizer-0.1.2-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a3f1f5712caa3d482a7dac69441681f13d92a23b1a892277a0da180181add3d |
|
MD5 | 0e89fbcd73597c4a3d68a999dbb33efb |
|
BLAKE2b-256 | ea3066fdfdf4c6d5133af06bcd100c5941f5bf463b76b1495c98db69dcbbcf75 |
Close
Hashes for toktokenizer-0.1.2-pp38-pypy38_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 32ef60966fe5faec25d40133757f7ae591e20e0ca1095b9039e9ff6283d19ac6 |
|
MD5 | 0c4b65d9d293c5a58372fef7eed41a43 |
|
BLAKE2b-256 | e52d81a05762dc6bff9e35ebe8d94df8c0dab49c44c9b62a3147fafaae05cd23 |
Close
Hashes for toktokenizer-0.1.2-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ea3d08af4d419d9edb5e8eb5d1225aefe7fb05a9a53f711680009064d63cbdf |
|
MD5 | fb40b2d4b01dbd86b55cfc661fcccce3 |
|
BLAKE2b-256 | 887bbca5284278bffff339ec2cd487c8513f909f32530c7b402ad680e5e556eb |
Close
Hashes for toktokenizer-0.1.2-cp312-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e62f755e123f834373081ef2aaa7b7f22a15dc413eeebf0921873c915c3a2e5 |
|
MD5 | 0f4c3a98ac56cb817cebefa658859625 |
|
BLAKE2b-256 | e29213f05d0a495e5a939363188c2ce2ab4335f68b86726c1e98b37998a12e56 |
Close
Hashes for toktokenizer-0.1.2-cp312-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1990f5c9a5ee6c0cb5ff2687c69b78c1a3844086e7cfaac159216186643933ce |
|
MD5 | 552b55b0d42bc32dabb10f899b93ba4c |
|
BLAKE2b-256 | 6b6442a62f0f493de20538215bb0fe3197bdeae909d59cd178cfe55d61227314 |
Close
Hashes for toktokenizer-0.1.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e16ba5c41076cdea547667b9472d1ade936ee89fb729bf16ea4609e3ad6eb40 |
|
MD5 | dd65d1c5b6856ebc168a56adb616da0a |
|
BLAKE2b-256 | cd4b3a0add12cd7686230be8aa1ff38cb81d5691720fd09e2cf154b45de14054 |
Close
Hashes for toktokenizer-0.1.2-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1245cc8a9c8a782c1932270a6564996e0f8d8eef7d4c7a16fac2f7a940f349d |
|
MD5 | c7a6f984c7a3cdf2ba4fedd878072d52 |
|
BLAKE2b-256 | fdc885032662dba74365f58c16a7ac1385606f1f79b97627fab8369d29b70155 |
Close
Hashes for toktokenizer-0.1.2-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d5525729be81f5890ca844ef52d51a60fbc88b1877145d72e005aa727a6d40ba |
|
MD5 | 5605b76afc97e3a58cf67ee979f0fa55 |
|
BLAKE2b-256 | 9973d670585d6024f3d6eaff6732fae8e2e7ecea08a83399d4cd4e530503a232 |
Close
Hashes for toktokenizer-0.1.2-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78567003d6c8fc392f5aae85b0ec680d1c40c556a0f2bbf8e3e91cafc9b6ea88 |
|
MD5 | 0cad584ccf80bf9cb8b512486075084b |
|
BLAKE2b-256 | 6bf1296e85b797216053fe0668edbdc64409c9efbda58139306dbc9bf7c0b449 |
Close
Hashes for toktokenizer-0.1.2-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3407470b5389c250c06735e734235e3eff5e1818374bf037fbb1a4745c63f0f9 |
|
MD5 | 1c135d734b1be18dae51e32b2bfaab19 |
|
BLAKE2b-256 | 330019f4ef4be922955369b8be06cd76343376b8e4050535fada9f67aa000ba7 |
Close
Hashes for toktokenizer-0.1.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 81eb5a815d6fd74cb4bff9ac8d750b3fca0babc75149602752b06dd794aff157 |
|
MD5 | 7d992ebca888a539ce92a2969e8c8cf6 |
|
BLAKE2b-256 | abbcd0eff8acba1531b6dfb8dc732c6e1f18b9437cf2745d0e2b94b1583972bf |
Close
Hashes for toktokenizer-0.1.2-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 696f2c4cf9269b3f53feec8db869c41246ea50a751c92d103c14db68809c6047 |
|
MD5 | a015c939a1ded4c7c33410320f356723 |
|
BLAKE2b-256 | f070530afda12fc329c0899787ce09b4d370158d5b05ae2c3cb40b3b8f1cdaa3 |
Close
Hashes for toktokenizer-0.1.2-cp312-cp312-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd0c2ce76855c3cba4af082e64ad8488a4b53f969e982895e8adbf3f14fb3475 |
|
MD5 | 13a62402f9e51548d4f5ca7a2bcdbff5 |
|
BLAKE2b-256 | 77108f8a4037d312b8708a77b8fa47def3e4d221b1fbf4009ef6b02dd3b0613e |
Close
Hashes for toktokenizer-0.1.2-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d6b9ba2bf58642e6ecf0cfc5550ca0606217c57028e1170c7b22c8433e2773c |
|
MD5 | b86936c8b6a13776f9676c4299d1a364 |
|
BLAKE2b-256 | d868ddf0ba18962bea4439e555d84f554b8b8109c011985350091aba43742a26 |
Close
Hashes for toktokenizer-0.1.2-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8efc0e8f6be25774da06f30762a4d1804153d982c567d1275cc40f53926d8de8 |
|
MD5 | aa8b64a1834647cbd11981fe7d9a0fdd |
|
BLAKE2b-256 | 5e7c48224a16e9c6c48c5a0a05e91d6104414ba23da27f711604754451241bd3 |
Close
Hashes for toktokenizer-0.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41a2f5990e9b8bb2746e22d030e3198e5c5a4f6a9535d7ba8469465a5493bbbb |
|
MD5 | bf8aa948bfa21c6573c44100e1d9b3dc |
|
BLAKE2b-256 | 51cc040c05f2b647e4efeaf01f68f95bfbf03f5a24ca4c7d452744c7ac6fd0ff |
Close
Hashes for toktokenizer-0.1.2-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 61a9d4f2b7053b7545b3eaf2c3264cfdf4a774e4a1dd1377b7146787dcd5b05f |
|
MD5 | 1c4dbef6b81d533a0f485625f4ef0924 |
|
BLAKE2b-256 | 6b7d147c715ff7c951daeedbeb582d0ee5ca496abf539b697e581fe1c7e8415c |
Close
Hashes for toktokenizer-0.1.2-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 942eb56bd6b0271cbf1b5fa853662def8e48534712b64c6bfcd43cf1451e05f9 |
|
MD5 | f545b002be293a0a0a1e2dd65bfca8db |
|
BLAKE2b-256 | 1b71ac7d4f3282ef76be5343cc5f233c68260b3a0e1e595b01b06652e599850e |
Close
Hashes for toktokenizer-0.1.2-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24f9c399909d29694a898a356c23af37c52909b4305f664aecd0e7537f0d5bd5 |
|
MD5 | d08034c76ee9033db3d16949b1371994 |
|
BLAKE2b-256 | 3e57ccd63fbcb8e0ef258d1b67c3a26f00ad834ee35c4a6084361a8fea913f21 |
Close
Hashes for toktokenizer-0.1.2-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f0218d8e58376e428bf5f929965e2476db94fd575454149bfe0df78a45e04436 |
|
MD5 | 38d7eb368719865632ce5db9413b081f |
|
BLAKE2b-256 | 5e22c653fed207b4e30bf9d3eef706fb6cd892f308217d62f8bbf72912a6c71b |
Close
Hashes for toktokenizer-0.1.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c94f91fc759b7f9a1c1e35ffbd6d1abf7568f4ebe8c9438214c334c1aa153161 |
|
MD5 | 45a32a8284052f8d4d902e89cef742e9 |
|
BLAKE2b-256 | 17f93676fd8965b4897490fcb0654a0c7cb536c3148e1ba310d5344d664ceaa8 |
Close
Hashes for toktokenizer-0.1.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2aa1ce08bcb279b84120ccae7e544d62f07c056bacfee88fb65a188d83a1fbae |
|
MD5 | 0ecb0ff58bd95d408afeb5f2a315d8f7 |
|
BLAKE2b-256 | c6c18e6ed2cf9e9b5f07680a5faf5fca99a4ed0ad581a35381dfa9d67b75fc59 |
Close
Hashes for toktokenizer-0.1.2-cp311-cp311-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a1f39c9f6f9c65f720e2aabdf038bdf668d23528e799a6e2a1cd5a500d067ba6 |
|
MD5 | 078094c7ee03143a3459ccb0d42529c6 |
|
BLAKE2b-256 | f66037ba4ceaab9cec1ef8081fb104dcb9d4938b1c37d0940f36c5f340148909 |
Close
Hashes for toktokenizer-0.1.2-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c2ddd1d734a7f0db4017d1f2c02f2e140ce505bbae010e8bfe4f5fe5b44c0a6c |
|
MD5 | 712b06e83b42a749c5375f8c8ac1751b |
|
BLAKE2b-256 | c6376de6da1858db2410eccf3158cc7852080b4f353170db59993ffe951d9f84 |
Close
Hashes for toktokenizer-0.1.2-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b39bbfa5b5862540e27e3e4b9a20c069df0c7487d8c274f13627c432fb45af52 |
|
MD5 | 5a8d223d7b6013fe8764055560477801 |
|
BLAKE2b-256 | 3973a5e0d6e70a9cd603965c331279edaa393825177cea135ac7d25a67a55586 |
Close
Hashes for toktokenizer-0.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f73cbf96f125077eb5122be0712c004aee339efa8628ed5c239af6044ea1ae13 |
|
MD5 | d6b6de1bb162beaf6a207b8d50c65f08 |
|
BLAKE2b-256 | 3291c2e6171c5c3833db697ac3e83ab9fc90f77f136425e0a50b6e90ba0b59bb |
Close
Hashes for toktokenizer-0.1.2-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ff8375e16c5b44c182094f3caff5d71306e0d59269865175545373b8a72ac695 |
|
MD5 | e9d16620f9eb42a9d389f49e2a2a918f |
|
BLAKE2b-256 | ec7523778b2634f075e081849fb1aca2d49c10cda746f64cdf5441a49189f01c |
Close
Hashes for toktokenizer-0.1.2-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59729590f46765eb75e3ad1d4649ab25557ef4559e2346db463cc11b50af06e0 |
|
MD5 | 2f59d2355d89533b2f3c9b9cc836f531 |
|
BLAKE2b-256 | fbd9bc3f87a076cc9b1a21a9ccd0c0d70f92c64f68b0bacfe67d6359e2f3fe77 |
Close
Hashes for toktokenizer-0.1.2-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2bfb42947519f6aee208dd593da3558dd53ec990ad4260d32d3be3f118463849 |
|
MD5 | b189a904f50f68ae0b477cb7aaa63b05 |
|
BLAKE2b-256 | d08321fc718128c026e8a807c332b8cfb9d58f5c8e89650cb78dabb324d8f5a9 |
Close
Hashes for toktokenizer-0.1.2-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12365bb1aad570f36e73a2b6dbe149e17d9acae1fb4767d890dcf0bb49db5a7e |
|
MD5 | c5f559110a1e53c6d2b20a43ad6a9284 |
|
BLAKE2b-256 | 7a4d625820874b980d29cdc27966426a2928e7e495081b7b9d71695feb670f16 |
Close
Hashes for toktokenizer-0.1.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 72e066ebcd766f403ac4044fa10fa486160aa3ec91bd854ee495a33162d8c738 |
|
MD5 | e531969521309ae5395f43ec2767cdff |
|
BLAKE2b-256 | 2f385fe9dec8c671f21d9e0b0dce470b9c6e4d2f52465e4f3e209db7ff299815 |
Close
Hashes for toktokenizer-0.1.2-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9dc8df35c6f4136133e3737ba77bf96768def6a107bbb915804fe02c6e072d18 |
|
MD5 | 3f0374e9903ad796c99a924825fe395e |
|
BLAKE2b-256 | a34042939ba11e0301ed4df31608203bc58ee64d1109332318ded5feeedadd3f |
Close
Hashes for toktokenizer-0.1.2-cp310-cp310-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b6a0747fb8af82626cb5ced1648374654aa21782991fd1c1f68c4af80f75a5f |
|
MD5 | 5cf7ba5c5f3617e3f632ba29ee6b4dc6 |
|
BLAKE2b-256 | 1d18e7b77f0a614612d50610d553cc59095d756bda41881cb9a8ff56219beb49 |
Close
Hashes for toktokenizer-0.1.2-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12431948ecf5434b8caeb8e43ed0b7c71378a5372cca554afb73d0db1f5de72d |
|
MD5 | 8366336e588ca979d839e58a10f0e5d9 |
|
BLAKE2b-256 | 213c670ca7160ecabfbf37e99b710ebb0664f8a6a01658f7427be2c9e8f7d3e6 |
Close
Hashes for toktokenizer-0.1.2-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 03233b952e17493944e64940eb8f329416d235609c6423c521ae984e23603fc9 |
|
MD5 | 69ddea75385debcea1f792558e861a22 |
|
BLAKE2b-256 | 6fc2fffad3b713ec37eb5ce51de1425d88667ccc763a71f0a69259ddbe46ed0b |
Close
Hashes for toktokenizer-0.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f90694e79c13ce0156c44ab949cd77b40e8aaf68986ff13dbfec47e6999c826 |
|
MD5 | 61f28c4b3bb11640676c5f4a409d7e2a |
|
BLAKE2b-256 | 0406fb5db35fceb3e266ff226622a9f2dfeedd89c49187e97ec41dccd0eb0149 |
Close
Hashes for toktokenizer-0.1.2-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2db21542678eb6c30ac03d0d6bbe4a0d36c3c5eb75cf17363674484e37c57c87 |
|
MD5 | 485fd8f88d7a9cb2cb9fc7569ef1d1f5 |
|
BLAKE2b-256 | 6a0cd6b0ce90ee85455fb9802ba76e92371701b9c3fface7772809546d4312af |
Close
Hashes for toktokenizer-0.1.2-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 668c3b095add1eecd4fc5e0da4abe7601f25c2579e0787dae5107f36550bc8b0 |
|
MD5 | 0ceb497dc913ed623dbfe7af06734d56 |
|
BLAKE2b-256 | 02f76b34b96f23b7f5aa750e41f1bc4c1fcf3eddeb12942f0a36eb86334af8d0 |
Close
Hashes for toktokenizer-0.1.2-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4857c0451d1194822763584fe5904e2a276a47709f3cfd8909eb59e9c6d20023 |
|
MD5 | b1328cdae1846525845d1f62b89d6fce |
|
BLAKE2b-256 | bdcaa95293af22c36a6d6dc2d18cc95b1937626f4d3bd32a468be7b5197649ff |
Close
Hashes for toktokenizer-0.1.2-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f326660ad595fde3b725253d6f883a86347ab1279628050d5ea24479d78d3aa1 |
|
MD5 | 864a5534670540397ce425f725a5c762 |
|
BLAKE2b-256 | a87773a5122c93c66d9ef2e318ac50598a58b7ef4e96ebf086ecad55e9f06b65 |
Close
Hashes for toktokenizer-0.1.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b8df0ec8516680f5a80e2037093be712fffd63d3fdfed0fac483e1ab5240bb4 |
|
MD5 | 989f9097536a12bbe4a0120fbf482ec4 |
|
BLAKE2b-256 | 687b036b15898c1c9134db162b3a5ccfdc9de7789f5b1b2838b82007bf12d279 |
Close
Hashes for toktokenizer-0.1.2-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f3f3b86cd5ca7163e75b859b4a9e3a9d57c5119910b3914e62a7797ea900fa23 |
|
MD5 | f5b1037cb246145df09debac22c63e13 |
|
BLAKE2b-256 | 92d7d0120b1effb5bb9de03a25455fd5b5c2cc0a1245e194d6003d98b09c13fe |
Close
Hashes for toktokenizer-0.1.2-cp39-cp39-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6794d28d06e5cc437c8f00117fdeb157ed4ea5440e6769c088e35f6d254f428 |
|
MD5 | 58f0a6ce59b432fd39846132689a5123 |
|
BLAKE2b-256 | a335504a2025f388c0f33b47c68ce52af0f95b8df15ab16aa2861e09f03d3704 |
Close
Hashes for toktokenizer-0.1.2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 56d6b46f138329ae3bf4ef93514bd9867f05b0ebeecc2eb0ca82da9bbbc7f96b |
|
MD5 | 9e2a8350b743c46c44b8ff417d8aca70 |
|
BLAKE2b-256 | 5f65b3dd34ed282809d0fd3c7df9fda6abd09900ab4fa80e5ea549a0548e461c |
Close
Hashes for toktokenizer-0.1.2-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66a5462d9ace494673cb9ee604acd26cee40859fdb3813d9da93c9ce9916bcec |
|
MD5 | 0c3927fad253d75e1363372cc021d98b |
|
BLAKE2b-256 | 828390299542459da80ee5ef44657a33454aace84092068b07cb001ffcff0cd2 |
Close
Hashes for toktokenizer-0.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31fe07b230498e38f7bbdf99545f08f958da72d57d1cd3fc405ccf5a8f2a8488 |
|
MD5 | dadaef45faef6125bb0b6965db5cb2e1 |
|
BLAKE2b-256 | 8cf99c28d39c114a00a5e121db6bc6f66a365d754efc88f7c298167f27179a45 |
Close
Hashes for toktokenizer-0.1.2-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a5acaa63b32eba8b99f631c56f2930d4730b16b0a3562a9e1e3a1b954d3408a6 |
|
MD5 | b6ceda772efb02d62f403dd92a3e6e5b |
|
BLAKE2b-256 | 46d3a41116975575669e047c1bd09a1a1c45f5f2c1915d754fd8c5c7b650c715 |
Close
Hashes for toktokenizer-0.1.2-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc2fe82b3fbc71bd65436cac79973975a20f6249801006fadfeda3f2bc27e4a8 |
|
MD5 | 7ecf93803daf21c58ddb9aec3ee59a6e |
|
BLAKE2b-256 | 177041ce70aa6b7991c11221fecfedc161235aff6fb976dc038dab7be8dff193 |
Close
Hashes for toktokenizer-0.1.2-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ab7e1749a13676d55a7e2df1bb1a6e433511f8192b980a31c01055f598407ca |
|
MD5 | 22ad41a4d12fbb45348023dfe464539c |
|
BLAKE2b-256 | 7ffee34ca29b2293706af359b7d4caed00bd87c9ec284ca568c5e2ec9cb96d05 |
Close
Hashes for toktokenizer-0.1.2-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9045d0a161df3a967a8f93b7d6db5bbf487f530882d5223c1508448c7a937794 |
|
MD5 | 2c01bbc754ec98c933db9be993c3ee31 |
|
BLAKE2b-256 | 639423d71e7fee2a88c4a2a155c0cc18993ce4372fccc091d13470e0f8f38a96 |
Close
Hashes for toktokenizer-0.1.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 80193fb2bd14503b2f295c9818bfbe0fad8731482ada5b756429dcba36d72164 |
|
MD5 | e254ac5cd2ce474d2a903d9f839a4b9a |
|
BLAKE2b-256 | af83be69fe57c8b328a033bbe108dad8666c03b50e41149f2c198c7ca6a25351 |