Phrase-first tokenizer for LLMs. 54-0 vs BPE.
Project description
Ion Tokenizer
Phrase-first tokenizer for LLMs. 54-0 vs BPE in head-to-head benchmarks.
Coming soon. Visit iontokenizer.com for updates.
Installation
pip install ion-tokenizer
Links
- Website: https://iontokenizer.com
- GitHub: https://github.com/creayo-dev/ion
- Company: https://creayodev.com
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
ion_tokenizer-0.0.1.tar.gz
(1.6 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file ion_tokenizer-0.0.1.tar.gz.
File metadata
- Download URL: ion_tokenizer-0.0.1.tar.gz
- Upload date:
- Size: 1.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.14.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
fb17a8519427730b93be67e318ea187ff31cfdb3c6d10af6576111914286b638
|
|
| MD5 |
3446794d916c89c72e7512202bdf23ae
|
|
| BLAKE2b-256 |
48b45e69b68c7dc4f87fb812c0899be855e6c24c0e313756cbd1513dbf914ab4
|
File details
Details for the file ion_tokenizer-0.0.1-py3-none-any.whl.
File metadata
- Download URL: ion_tokenizer-0.0.1-py3-none-any.whl
- Upload date:
- Size: 1.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.14.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1aa6d9977139d39c38056d5f0b7fbf85c47c2889ff03e0897f8a2976f5a93efc
|
|
| MD5 |
8ac6ef8a6f2acd4a9d22f07ff2c9746c
|
|
| BLAKE2b-256 |
976d463624a0f92a1b1b18e9c5dea7087fe33f5da858e8372fac71f0005a5757
|
