Prune unused tokens from LLM models
Project description
Remove all non-English tokens from the model
pip install prunetokens
prunetokens /path/to/model_directory_with_safetensors
Requirements: safetensors to read and write file. No VRAM required.
| Model | Before | After | Saved | Vocab |
|---|---|---|---|---|
| Qwen3.5-9B-AWQ-4bit | 8.5 GB | 7.0 GB | ~18% | 248k → 146k |
| MiniCPM5-1B | 2.1 GB | 1.9 GB | ~10% | 130k → 94k |
Pruned models:
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
prunetokens-0.1.3.tar.gz
(5.3 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file prunetokens-0.1.3.tar.gz.
File metadata
- Download URL: prunetokens-0.1.3.tar.gz
- Upload date:
- Size: 5.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
5556ca4ac76bb2b4f0d81476ab9497b20902983c1fce30ef86617ecef74c196f
|
|
| MD5 |
18ff82963588aca2836bd644c9fca7bb
|
|
| BLAKE2b-256 |
3a65bb4fc10e2ea068c2619391134d5daa72faeca7abe95eb3ac02b30f2f3fba
|
File details
Details for the file prunetokens-0.1.3-py3-none-any.whl.
File metadata
- Download URL: prunetokens-0.1.3-py3-none-any.whl
- Upload date:
- Size: 5.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
02b540423d69b5fc7fdf9dc047a06b2ee54c6bad9207a7b43710de03b7f569d2
|
|
| MD5 |
0b0c8b4647c066d7f0ca25ac9641e0b5
|
|
| BLAKE2b-256 |
4e4da5790e5e3c528af43b720ebd5a41908f53208e00aa82bd8850e8b70166a2
|
