prunetokens 0.1.4

Prune unused tokens from LLM models

Remove all non-English tokens from the model

pip install prunetokens

prunetokens /path/to/model_directory_with_safetensors

Requirements: safetensors to read and write file. No VRAM required.

Model	Before	After	Saved	Vocab
Qwen3.5-9B-AWQ-4bit	8.5 GB	7.0 GB	~18%	248k → 146k
MiniCPM5-1B	2.1 GB	1.9 GB	~10%	130k → 94k

---

Pruned models:

• https://huggingface.co/albex123/Qwen3.5-9B-AWQ-4bit-pruned

UPDATE 0.1.4:

Model	Before	After	Saved	Vocab
Qwen3.5-9B-AWQ-4bit	8.5 GB	6.8 GB	~20%	248k → 137k
Qwen-3.5-4B-AWQ	4.1 GB	3.5 GB	~14%	248k → 138k