Library for manipulating the existing tokenizer.
Project description
Tokenizer-Changer
Python script for manipulating the existing tokenizer.
The solution was tested on Llama3-8B tokenizer.
Installation
Installation from PyPI:
pip install tokenizerchanger
Requirements
- Python 3.9+
- tokenizers>=0.21.0
- transformers>=4.47.0
- tqdm>=4.66.4
Docs
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tokenizerchanger-1.1.0.tar.gz
(12.1 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file tokenizerchanger-1.1.0.tar.gz.
File metadata
- Download URL: tokenizerchanger-1.1.0.tar.gz
- Upload date:
- Size: 12.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
91ac30b40930e8ecdcf855dbd3b94ed4bbf9866191ce010740f361845236fe11
|
|
| MD5 |
782f3ef243796672bb396361c49772b5
|
|
| BLAKE2b-256 |
7d49a4692aaee6970698babbf0c773a84366e0b53694934399a0f9e44143783a
|
File details
Details for the file tokenizerchanger-1.1.0-py3-none-any.whl.
File metadata
- Download URL: tokenizerchanger-1.1.0-py3-none-any.whl
- Upload date:
- Size: 12.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d2043608193fa6f741e2be6d4765bd8469dcdb25f022976d85d2c907f314453b
|
|
| MD5 |
ec76b1dc28ffdd70ace24d598cf34e8e
|
|
| BLAKE2b-256 |
8e34690834be34735fd9b2a14cc532ef3adaf844bd53b8fe1973edc206b09dec
|
