RWKV Tokenizer
Project description
RWKV Tokenizer (WIP)
A fast RWKV Tokenizer using Rust. This is my very first program using Rust, so there are still many things to improve and to fix :-)
Installation
You need first to install cargo (rust compiler), if you don't have it already. Cargo will be not needed when I later publish the whl package in pypi
$ sudo apt install cargo
Then install the rwkv-tokenizer python module:
$ pip install rwkv-tokenizer
Usage
>>> import rwkv_tokenizer
>>> tokenizer = rwkv_tokenizer.Tokenizer("./rwkv_vocab_v20230424.txt")
>>> tokenizer.encode("Today is a beautiful day. 今天是美好的一天。")
[33520, 4600, 332, 59219, 21509, 47, 33, 10381, 11639, 13091, 15597, 11685, 14734, 10250, 11639, 10080]
>>> tokenizer.decode([33520, 4600, 332, 59219, 21509, 47, 33, 10381, 11639, 13091, 15597, 11685, 14734, 10250, 11639, 10080])
'Today is a beautiful day. 今天是美好的一天。'
Performance and Validity Test
We compared the encoding results of the Rust RWKV Tokenizer and the original tokenizer using some parts of English Wikipedia and Chinese poetries dataset. Both results are identical. The Rust RWKV Tokenizer passes also the unit test that the original tokenizer provided.
We did a performance comparison on the simple English Wikipedia dataset 20220301.en among following tokenizer:
- The original RWKV tokenizer (BlinkDL)
- Huggingface implementaion of RWKV tokenizer
- Huggingface LLama tokenizer
- OpenAI Tiktoken
- The Rust RWKV tokenizer
The comparison is done using this jupyter notebook in Google Colab. The Rust RWKV tokenizer is around 7x faster than the original tokenizer and 3x faster than OpenAI Tiktoken.
Bugs
There are still bugs where some characters are not encoded correctly. The bug have been fixed in the version 0.3.0.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for rwkv_tokenizer-0.4.7-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2df1acbed5524339a8a30d020e3776fbcb1e462bb7e4ac7a3b5fb0cd47d07cf5 |
|
MD5 | e5869fc1fedaf43863b7296ae1f58182 |
|
BLAKE2b-256 | 053e1ab4ceaef7cacbe496eabb55d7dd32999fc7e01a1adad7102633f63f5747 |
Hashes for rwkv_tokenizer-0.4.7-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7918ea5be33ff205f88587e1786c5aadde7595df463e5e15cd0dc7956b8e5130 |
|
MD5 | b494bb80746538e4410abb550dc2e457 |
|
BLAKE2b-256 | e8e8b1d2ea93ded502adc31a0b40334b8d77eea8d4c152c1333c68d0bd6b0824 |
Hashes for rwkv_tokenizer-0.4.7-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9507225eae52791267655040f6b614db11702297c186141793ffff377ddb241c |
|
MD5 | 195dc31aa433c63be993e526dc25df63 |
|
BLAKE2b-256 | e327f017b15018861bc4c662e180da251762c689231f1e21b9e8192ab25cb502 |
Hashes for rwkv_tokenizer-0.4.7-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1983c25bb8555d88955994bd59b1178ca3a25b6cbb1cb630817809dde58f327 |
|
MD5 | 5be086d8cee171e393ebccd61a34ad87 |
|
BLAKE2b-256 | 3b8eef6fed03a63421a88a253333df58c0853c76474a54181de2d06526e27f04 |
Hashes for rwkv_tokenizer-0.4.7-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d0d34d6373ead3bce90ca4b00247867822e0836c6aa93d3b12e96ae8feacc743 |
|
MD5 | 0b449ad81aae63aaad511e25818fd4d1 |
|
BLAKE2b-256 | cff5ec60d2d8a8236bc66afdae0c3455bb8b3c991fbb34459c0900cdae41b394 |
Hashes for rwkv_tokenizer-0.4.7-pp310-pypy310_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cff05116251b94b05ef82affbdd4b257f707a870f1a1ea86107462ad34eafa42 |
|
MD5 | 8d1dcca19e74b31d4eb8ce0fe443087b |
|
BLAKE2b-256 | 01ee4637e5902fbc5d92967925970d30ae599158fe306b867f5f634aa73fc0c8 |
Hashes for rwkv_tokenizer-0.4.7-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76d359657895a5f77b33e0bbb373556d4d9b7892bf4682b3ef5c64176264b6c6 |
|
MD5 | e6e9f6a345abdc2a30b7e437037835ca |
|
BLAKE2b-256 | 8cada9fe51c3582ef785b63496e9f80d0b7b2ef95af06887d8d00ad6585de57a |
Hashes for rwkv_tokenizer-0.4.7-pp39-pypy39_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fdeec5b888e7eaa208cdbbc1c9c082ce27fd87571b36db1d2f463c691e7aba9d |
|
MD5 | 9eee7711d0fec4afb6a514577dd76731 |
|
BLAKE2b-256 | a4037a2c0b2afec08eb607f9b0111ad31ecbe08590cde843e9565d0e383ae8a7 |
Hashes for rwkv_tokenizer-0.4.7-pp39-pypy39_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa4ea0dfc6615d0bfb4972ba30a91f56f791a10dcb1b7872c1474b1d2c537180 |
|
MD5 | e9069a244adfe9610485093ae70e2605 |
|
BLAKE2b-256 | 2e55c169057fd570df7373f8104a7c358629d03bfe6253dd0f32e166386aa022 |
Hashes for rwkv_tokenizer-0.4.7-pp39-pypy39_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1859b4a969f62434de96c4f45ec6418580175ee052f9012ee33763ebabab0747 |
|
MD5 | ea4dc4fe9c1554bd09080a1f4a239e0a |
|
BLAKE2b-256 | 6c05490bb7afdb55c8349798f0aef3eb1ce8cd69c17c32c64b480257be57729f |
Hashes for rwkv_tokenizer-0.4.7-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 705bf093a46dcbb52fe14748f355080e6279025c594c657e8163a2c390f4cd20 |
|
MD5 | 3e9aabc03a425481762fab4dd3ceebee |
|
BLAKE2b-256 | f8a0e4e1f39b68771ef8c720ec1da4cb202619823c43ae660e5d2c9d5060d5cb |
Hashes for rwkv_tokenizer-0.4.7-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97307104226a06f1e5c3995d99228c6ce3ce165c78efe4d1d7b13f90041860fb |
|
MD5 | 91d4ca04d738ef693d5c7e69f5560d56 |
|
BLAKE2b-256 | 035354522139b143332f3b5b410b2bf39220c35e71d07d6e8268676aad57d924 |
Hashes for rwkv_tokenizer-0.4.7-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7378057272e827823c45f42e7b37566db7c75c3a4ce2fc3d14c2a6fa8ef9a8ab |
|
MD5 | 9f41a551110c607a5f1993a6abfcb0c8 |
|
BLAKE2b-256 | 5d8834898492a019adaf6d3732564897b75e18e82a4b5ad047b060fb78f9e2af |
Hashes for rwkv_tokenizer-0.4.7-pp38-pypy38_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3afd08b97a20526bcc20f06ce8bf62f7c3b2d1415cf2af61df193cb1554413b8 |
|
MD5 | 6ba718163492c38cd3324c0b8a3ce55e |
|
BLAKE2b-256 | 7332c1cec47091c1bdc5553b2f0dbd760499b4fbda8ea9c6aba9dcedd9adf427 |
Hashes for rwkv_tokenizer-0.4.7-pp38-pypy38_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 649a82148f8dd58f965efedd6a79f3945432c79275cee028123daf9fac1ce7fe |
|
MD5 | da340449e86616cd8bf4e3ab60bd0a9a |
|
BLAKE2b-256 | 08bea06789a5392abf1f96afcc60e880834c1b8772be0e9e67cdf970e1e3768a |
Hashes for rwkv_tokenizer-0.4.7-pp38-pypy38_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab218a6e34c0406c614e9a5fc7df9b9bf01dcedf61d10052079b123c0457d146 |
|
MD5 | adde1f1633f68346366f955831350b30 |
|
BLAKE2b-256 | 359a4ee1316bb05a6d2da8da326549b1e34f1ab17c791f96c2a9354089fd9ff9 |
Hashes for rwkv_tokenizer-0.4.7-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c17525680d13bf155e92c1269ec547b9f2729df29cfa62b25ff33979c914608 |
|
MD5 | ad42973c8b5fc440f5759c10772e3cc5 |
|
BLAKE2b-256 | 1f0f8ec02a7e8474a48c50a0b417ae661d922930c2a65409585fb80c15c3ea29 |
Hashes for rwkv_tokenizer-0.4.7-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1f7a14ef131b5e5044f1f421e9d21713002c3645455e2fd8528e543b4ecbf914 |
|
MD5 | d763b290c851a0dd6277108c7f267a49 |
|
BLAKE2b-256 | f556d96a5bad6f4bf808310f72ec00ac56b0270dd28d76fba7df27b06386b880 |
Hashes for rwkv_tokenizer-0.4.7-cp312-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c40fa1d3abb42b4de1f9148f5b2d21bbbec9d8afb28a89f57282f1037dd4afb |
|
MD5 | 5c92eb79de442732377407922ccc4c00 |
|
BLAKE2b-256 | 00465e0153c4109147f4dbbb48d230dcd820b541d41ca7b7cd5f25036aea10a3 |
Hashes for rwkv_tokenizer-0.4.7-cp312-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87b82269c4e434cb747aff91e85e849ee2592340bbf8c468c2522e63fa5eca9f |
|
MD5 | 54b5cc313ca6ddff4fd5806257e0bd90 |
|
BLAKE2b-256 | e9ab0e031d90ede2509a9c7299ca56e02f7478b32c13d41c4172923b50b77353 |
Hashes for rwkv_tokenizer-0.4.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f99abee5f3b4a82df54236ddd1d35ae6c9d948f711e98f319f664dd67f541bb4 |
|
MD5 | 341f9e34c95f165d0d0fba199d922d54 |
|
BLAKE2b-256 | c0a7247262f9a781bf48eca4c6a5bf97d51cdfb8ef52e6a976dcefe65448da9f |
Hashes for rwkv_tokenizer-0.4.7-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 27eab481fd83eb47a74db78e4bf1aa9c1a4f8b9bf431aeeb9bcf087fafe9d488 |
|
MD5 | dd695fc44a4739e18cd18cf2b4d1bfeb |
|
BLAKE2b-256 | ac2388789349b2ba946c28abaf559d48557e2df66809c17678255b8c228169f8 |
Hashes for rwkv_tokenizer-0.4.7-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 37adcdfdcdd0ab14df31271f02afa14883cffa9ee12f94e34cd967a6f2d96607 |
|
MD5 | d71e1d999c1cab8d1a6b888dd719734e |
|
BLAKE2b-256 | 62e6924d351195934eb95772c4df702e1412018a03ca45123ae95332d25d6ee9 |
Hashes for rwkv_tokenizer-0.4.7-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b025bb3bbab1656380ff92449085e603147464874a9fc094bca957a7a8d2ee1 |
|
MD5 | 4825ee528eaf6b7fa1e75a30c86f2348 |
|
BLAKE2b-256 | f70a0ab774019a15a9db5a39cd8f04fb9e54c6851301e4218bfa4c54ea7b21de |
Hashes for rwkv_tokenizer-0.4.7-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d280cd1d2241c26541cd62b0e69c7819532a55d7b69fddd667e6b41900be5692 |
|
MD5 | 19995a5b7cf145b953c8ac100ddb8f1e |
|
BLAKE2b-256 | 88dfd1f2ba8c38bcc9207a855173a76fb7e4e5b2aec77aceb3238b06c4d3db8b |
Hashes for rwkv_tokenizer-0.4.7-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b77940bae61c23522503062bb57d195a1538b392f9cadfeae0c26d7ef68aed5 |
|
MD5 | 40f514b8c3f33d6d050d34a0fb830b18 |
|
BLAKE2b-256 | b0fdfdec4323ac4beedbf5c5a2bc167f1ffdebc78d3e100e8a5a018c8e2c3b34 |
Hashes for rwkv_tokenizer-0.4.7-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29a0f1150ec5888f4e7f95e6361532612e430e087a0ab91a994aa6381d0a23f9 |
|
MD5 | 0abc1d7b46e94eb522c671bb740fd4ee |
|
BLAKE2b-256 | c1c8cb0f6dfb2ccff23b7b24d2eb1bf2159e0e0028947f1c07ea57b19c22adb7 |
Hashes for rwkv_tokenizer-0.4.7-cp312-cp312-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bcf72f050fd884f1ba700e5317caf55f8b36cf08f20f14354290ee91463a38ee |
|
MD5 | c1b2df059a69b728b54e723f3eb30c51 |
|
BLAKE2b-256 | 59ee143e4290a3d568ed2c6f068dfcdce1c28f0b84097ee520158ac8745ed9de |
Hashes for rwkv_tokenizer-0.4.7-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3796218c7b618f2d0e8acc5e8e20f60302818c622bbe45729481a62894cc83c |
|
MD5 | 99f225f250275884ee4a053be8f825f4 |
|
BLAKE2b-256 | 31a24df688776c93abf14dea8bb15ab0df9f3685a331e8943c4b16ed0aedd494 |
Hashes for rwkv_tokenizer-0.4.7-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ece40b90835e8e94d5ee6dcb9c1e454e0cae00cab50d8318fd950ea60aad2450 |
|
MD5 | 4cd9c966da475cb182ba6d6103be367c |
|
BLAKE2b-256 | 8fb87cb0472960e6e9536dea7a97825d6341754a183cc0c78728a67bf6d2f189 |
Hashes for rwkv_tokenizer-0.4.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43d67738ff5fe4dc48b99d181c1683018aec18aada2fc9a28c6c6bdc9f97dcc2 |
|
MD5 | db2b3185868112b46783e53aeb194b34 |
|
BLAKE2b-256 | 2bacf80d74b658d2b0d261ec03b57c13458686c946dc00289cc8724ca10ff712 |
Hashes for rwkv_tokenizer-0.4.7-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5b1bbfc83ae296b6dfd479ec7200e058e3e6bf1d1c959ab49e6c2eba2ed7704 |
|
MD5 | 1d7853a41da7cbe1159a2ce395042399 |
|
BLAKE2b-256 | d9abc14e145da4431de7ead6643280166cc0eb707848c2f738ac02c431331b01 |
Hashes for rwkv_tokenizer-0.4.7-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 003b5b190cdd1b8de1321c58be290d00a73332cfb44058fb2f96b683cbfa5c32 |
|
MD5 | 28aab49f544d82177e9c1e4db6b1d142 |
|
BLAKE2b-256 | 64e793356b34db14a4785ad54fb66d6d0b7c1868acbb8e91d3de0706cc391f2a |
Hashes for rwkv_tokenizer-0.4.7-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c290e8e069213678e36a19ab686ba5c16d965e5b5294d3096393e240b9c0593 |
|
MD5 | 1c34ee57a1a483dedaa4bc31596ac50e |
|
BLAKE2b-256 | 2231f7be98e2eec69e08f59160b6bb3f764a16ec8074bade9809540fa3190939 |
Hashes for rwkv_tokenizer-0.4.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c04bb1789f082b1267b92c413eae0dbc2b6c4e7df0d894fa161fe0e224a009f3 |
|
MD5 | 256488c7073135da954fe92828f661b5 |
|
BLAKE2b-256 | d92e4b3cb548c6107f2c5cb99847c6f3e333c7499e5e999d1167a20a58c458ec |
Hashes for rwkv_tokenizer-0.4.7-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 40f99f4dfdeab5c903e776710b71f90c8cbccdf4b08ddb1cca5c0bae64f9d54d |
|
MD5 | eb5a6fab69cf5a7f3af54eba3ca8ef0b |
|
BLAKE2b-256 | 6970964a8dee5310257d5092ab7dfb85dc6e9fbf67ca32db5750800ae87af305 |
Hashes for rwkv_tokenizer-0.4.7-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f75680240fe85e423b309a709d239caaf7f2b5aa1a665cb0dc0d205b01775d28 |
|
MD5 | 29c07265c0ad2f5e2a62d491bdcf19f2 |
|
BLAKE2b-256 | 749bf600d0d2bd369337f4f74908f80576beb3f6d2d9f13dc6eb22ce2f2e74fb |
Hashes for rwkv_tokenizer-0.4.7-cp311-cp311-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d7d06f95ed80fa0b9e04b7325e1450c7b7a86dd2be1bd6bd2794bf16c737e5a |
|
MD5 | 2908c190c59f59bc7b4c90dc17c0d02c |
|
BLAKE2b-256 | a3647f90aabbd3bb75815e0af84b47d9045a229935a392e230e48ae7c4978270 |
Hashes for rwkv_tokenizer-0.4.7-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d07d9753e38520aa9c5da582e59b00327eb720fabac7186782d935ab65b8d3b |
|
MD5 | c9d4f274fff86b92fe272b49f95948e1 |
|
BLAKE2b-256 | 4bbd0dd89b0c4c4992df119d4ded5e20ea9a48021b9e2414f340ef2634c105d8 |
Hashes for rwkv_tokenizer-0.4.7-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c343204ee9c6fa2dac8afda0bc9edb58847d3e1dbbd422da387e614b37bd2d7 |
|
MD5 | 3c0729e83321869ccc091bbc3d5eeff8 |
|
BLAKE2b-256 | 3776b5ab403939afe7fc1f3ba9154a7d1c51df431d17fb50f78aeb45e9ed10ab |
Hashes for rwkv_tokenizer-0.4.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78b9a4ed9f7e1a6bf5687eb3c4cd0760992bd3231eb8a808910dcbe3b334170a |
|
MD5 | d73eaffb2dc9412a36680efa4325c925 |
|
BLAKE2b-256 | 3a72ca8ced4643b853038f51c05108f7ce941399b0378f7730d7b931092470a5 |
Hashes for rwkv_tokenizer-0.4.7-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0525b43468e319ac6439e3163d9c3794233d60bc138249000f204327f98c75a3 |
|
MD5 | 09c21278cc4ae06210a05fd9b55c773a |
|
BLAKE2b-256 | deab159188d708eb30a9465d8f14ac9967cbc4cf7968e7d17d7d829dbb87e437 |
Hashes for rwkv_tokenizer-0.4.7-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e2e69a1d9c11033bdadc61baeb498e72b305d5ff8eed62527c3d970b52dc2b8 |
|
MD5 | c670072b90c28627fc4823b2974a1034 |
|
BLAKE2b-256 | 96f0938f09a3a4f21a94b5706832dfef5dd3b6bd124b65577f51a827a15b7e59 |
Hashes for rwkv_tokenizer-0.4.7-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f07d0a1d091513269ffa84b399deb0520b606bb583a34b2edd30b5061554817 |
|
MD5 | 4147002626e444d3090c6fdaa61e2d3d |
|
BLAKE2b-256 | e25e9e9211cab35d6c51feea3f2cbbba3f587106cab58f4e8fad6757955637d4 |
Hashes for rwkv_tokenizer-0.4.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 26dab1ba2e6baa02006531c22629d5a176493c3b8813766b9deb379cbdb63c3e |
|
MD5 | 0a5fd0cf2e6dfb7f79bda012a2a7d409 |
|
BLAKE2b-256 | 865e74ed5145e4234509c842199c0330a6226a18f04e6abca34622af46b8c695 |
Hashes for rwkv_tokenizer-0.4.7-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d07803d4348030c3d16ce0f74d2f5008b7a7fea0fd7a716fe0e09f00b2752300 |
|
MD5 | e312fc804cb51fc8a8bcaf40801e91e4 |
|
BLAKE2b-256 | 08ccefac8756b0c530a773a2cee23a5e1f933cbf2da3cafd87bea6cfcc424bc5 |
Hashes for rwkv_tokenizer-0.4.7-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e65e47a9d47d9dd3c2a794da65f04f42bd1accd383fd10a164dbe423dfe34333 |
|
MD5 | c08bf3eff5cc484b9b88767951110c9c |
|
BLAKE2b-256 | a841a3f70f31d74fd528defce57134700ab6678f449a25929c1a14a873dd13fc |
Hashes for rwkv_tokenizer-0.4.7-cp310-cp310-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ed13bcf605cfd5f11f65df893f7a279d483f32a385f57636f3c9b0751b616b14 |
|
MD5 | 3313607b3f0152fd58fa4b48d870ec4e |
|
BLAKE2b-256 | a0d62b54b4724a1591297bd9e6031b8f4d897f67cf5be4af2468dbee1c41de72 |
Hashes for rwkv_tokenizer-0.4.7-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e14ef93e357b01165b5025979818361f62be06d37db17d23804f249d38c9ec83 |
|
MD5 | ad9b3b485bf6d54adf9f0a915d627dca |
|
BLAKE2b-256 | 60e040fe4770996d91b1a890070db5bb625b531837cb55bdfe1ab5c717c5cf4b |
Hashes for rwkv_tokenizer-0.4.7-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0833fed168cdab5b980ff455b12a89d2e8c16f6377f37b02b381f0fe98b4df2b |
|
MD5 | fbe6e3ad2d3d23d85e3575982b178417 |
|
BLAKE2b-256 | 3eb9077a0fbb239f730f6a6855dae077f87f6d4e481c887124a5756b4a9582fe |
Hashes for rwkv_tokenizer-0.4.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f316886b680d5792056bc037b6e4e266525017203ef30419ea0b33cd5b17e16 |
|
MD5 | 09c176efa0c4a7fcf602811693a31f8e |
|
BLAKE2b-256 | 0cd9c6e6a92a81f5affd376086d0ba1ee4c7aaef91ab659fd134a4a80a6b91c4 |
Hashes for rwkv_tokenizer-0.4.7-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93160e192da00ba99d0b9e12f324be588ef25926e941f355264f0cc69e4eb80a |
|
MD5 | 18dea3fd56f4557f60c539abfe6f7134 |
|
BLAKE2b-256 | c9d76deff7673159d44e9aadb6d7aa8c1dd4a0ca212ca0a600e87063544d2704 |
Hashes for rwkv_tokenizer-0.4.7-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5f9139df75615901eecfa437dba76a1a760abd8e361b89460a2166de3f1874de |
|
MD5 | 14a570c9a95de30deedf4f444f472662 |
|
BLAKE2b-256 | 99a722a9853b87c51e6ee1a4352ddb6663809d4f6639fdce1d8f4e8d09f036c8 |
Hashes for rwkv_tokenizer-0.4.7-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 361957e5a00475734cf53de1f69103970cd0ab1acb934c5aa7348ffcb3cea6ae |
|
MD5 | f1b983896edeaa678526914dff251c17 |
|
BLAKE2b-256 | 7087abd9fec64b4058417e74d9703885475e049650949ccb5f8c6cff2a918ec8 |
Hashes for rwkv_tokenizer-0.4.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d33685b2e0dab36f2a6d35d5f96474c5958bbf9f62d8a8394559121e0f07df18 |
|
MD5 | 3e0d493c23bacdb9a1117781096126a1 |
|
BLAKE2b-256 | 0e7a5138d152eb047d957edf7092b879846424bc29d79a981637804522bee564 |
Hashes for rwkv_tokenizer-0.4.7-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9cc84af50d8c7970598436a7a4efcb97c29c59749f9c4491f0626787e848f27b |
|
MD5 | 600f15d641248832cbcedd7e8c8111f0 |
|
BLAKE2b-256 | a3396fabee329077ce85495ca5ff30cb00129c2e9880e44d0023e6b116b0c178 |
Hashes for rwkv_tokenizer-0.4.7-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0fd84561a2b23dfe891fd7f70ace121886d06c2f2aecd2187472eb250d466be1 |
|
MD5 | fadf0495ece8a29ec723944eefc4bc5d |
|
BLAKE2b-256 | a3f46ef176a453ceed57a95b416276a2ac49b3dd25428385230b107990b51a41 |
Hashes for rwkv_tokenizer-0.4.7-cp39-cp39-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77ea5c606f54d5a71935b2448471726433106e29d99a6cf6737e9b46759fc06e |
|
MD5 | c556b774e3502c23df8ed8c3354c8de4 |
|
BLAKE2b-256 | 10aa1b92a35bd7c0a21d0daa1c22c0364006b5a5d0a91025bae707146dcd1e11 |
Hashes for rwkv_tokenizer-0.4.7-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c69750eaab0f02130e735023dd7bbdcdf0aaf62686e4796f13fda68704b77bed |
|
MD5 | 2f1b529e4c1196853408fada94be751b |
|
BLAKE2b-256 | b26c646e132307169b5e3e36232c24fb9b507fdfbadcaf2ae3b34bbf67d7cabe |
Hashes for rwkv_tokenizer-0.4.7-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2f47a147c4a95facc81520c5df910e902d24e91474b64ec11c6442b45487b23 |
|
MD5 | 3d150736a6a5fd26ee984765fbd317e0 |
|
BLAKE2b-256 | a8a779dc306aab243b5bc25a337fba3ff4f18bda169edd1f24918b8f150b3ba8 |
Hashes for rwkv_tokenizer-0.4.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3d383892b4616c211cbaf3070b6e9c4254e71c145185d9675e1e0833c2c5cf6f |
|
MD5 | 4099b2ddd48d04c9873ed4106dda870b |
|
BLAKE2b-256 | 312955aae2a78910e38b2242e6a92dcc3eb8930d3fc7e6417217a050dda2e0d3 |
Hashes for rwkv_tokenizer-0.4.7-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2dd0dcd2a3ad86761b262473c5c3530b27fbcc724e5e4c20e60ed4d1119d5149 |
|
MD5 | c5ec1e32d8434e6a83d7996456cbbad8 |
|
BLAKE2b-256 | e842e35fa3959a0df8ecfbd50003b64cabb385b3358d2c6fd43b04047c8cecec |
Hashes for rwkv_tokenizer-0.4.7-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 213bf49225ec52ed62546f1a2c58135b9c4314d0e402db38b5d3b157b827e696 |
|
MD5 | 3603781a3054354b1243c68a28d63d8f |
|
BLAKE2b-256 | 1da7c4f22c68fe11ad1e0c8b977f04ac1cfb75b3b8e5e22e036d5fce0f896a89 |
Hashes for rwkv_tokenizer-0.4.7-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4dbe4cb7fe530294527a1b88b598b18dbcec91550b770f985c601fffa2328ba6 |
|
MD5 | 7724c157978b9d331daa198dc5923ee5 |
|
BLAKE2b-256 | af1a021109b6fe9e9823add512084380007b699edc66241b89713ee85397e0f1 |
Hashes for rwkv_tokenizer-0.4.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ad84e3c957e8a53e28e4a692f07813c1e5405843df334d595f66bec7885bf436 |
|
MD5 | abf84bb4a131d692f576fd102e7f29d5 |
|
BLAKE2b-256 | 9dff754c5bf6472b35b119342fc5092f4e5c4364959beca0b0a1bcb4e202cf2d |
Hashes for rwkv_tokenizer-0.4.7-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f11cec229d1cdd19aed30c21361644ca27545250abbcb48249935bf3db657ee8 |
|
MD5 | 70d5c43707fa69bfef3d7e4735f6dfe2 |
|
BLAKE2b-256 | f39e4f7cdec8f4f35f4fef9b661d4f7d9525b356cadbd80e2024b368683c5f5e |