Byte-Pair Encoding tokenizer implemented in Rust
Project description
smoltok 🦀
Byte-Pair Encoding tokenizer implemented in Rust.
See GitHub Repository for documentation.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
smoltok-0.1.1.tar.gz
(208.1 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file smoltok-0.1.1.tar.gz.
File metadata
- Download URL: smoltok-0.1.1.tar.gz
- Upload date:
- Size: 208.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.10.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f79060d53b9396bebfa40fe958edecf1c93a689fe21e16d1d076e40d657dd481
|
|
| MD5 |
847c9802f73c35a8440a5338d6a104ec
|
|
| BLAKE2b-256 |
231e4b2328d66db8b52a61ce5881155e9bd652a17dd13bffb434c5fd5f4b64b1
|
File details
Details for the file smoltok-0.1.1-cp38-abi3-macosx_11_0_arm64.whl.
File metadata
- Download URL: smoltok-0.1.1-cp38-abi3-macosx_11_0_arm64.whl
- Upload date:
- Size: 970.2 kB
- Tags: CPython 3.8+, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.10.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9a7b6d2a5b20f92f92e39d2f5c412a73571a63b789c5347019bdd727f44b2e29
|
|
| MD5 |
1e15df1df5c24237655a822c98a9ba50
|
|
| BLAKE2b-256 |
c497f80f0a0dc9514afb7fbb6e9f2133db6e44223234e7c89fcbbf3f2737d2f2
|
