VocabSlim

A package to reduce the size of 🤗 Hugging Face models via Vocabulary Slimming.

Large language models (LLMs) increasingly adopt extensive vocabularies to better handle diverse tasks such as multilingual processing, code generation, and function-calling. Recent research highlights that expanding both model and vocabulary size enhances performance, following scaling laws with vocabulary.

However, larger vocabularies come with significant drawbacks. For smaller LLMs designed for specific tasks like English or Chinese chat applications on mobile devices, a large vocabulary can be inefficient. It consumes computational resources as the model must predict token probabilities at each decoding step. Additionally, token embeddings occupy memory and lead to larger activations during inference and fine-tuning, exacerbating the issue for smaller models(e.g., 26.53% for Qwen2.5-0.5B).

For instance, the Qwen2.5 series illustrates this challenge:

Reducing the vocabulary can significantly shrink the model size and enhance memory efficiency for both inference and fine-tuning. This benefit is particularly pronounced when models are quantized, as most quantization methods avoid quantizing token embeddings and the language modeling head to preserve accuracy.

Installation

From PyPI (recommended)

pip install vocab-slim

From source

git clone https://github.com/Andy1314Chen/vocab-slim.git
cd vocab-slim
pip install -r requirements.txt
pip install -e .

Usage

Command Line Interface

vocab-slim --model_name_or_path Qwen/Qwen2.5-0.5B --vocab_size 32000

Python API

from vocabslim import VocabSlim

# load pretrained tokenizer and model, train new BPE tokenizer
vocSlim = VocabSlim(model_name_or_path="Qwen/Qwen2.5-0.5B",
                    save_path=f"Qwen2.5-0.5B-Vocab-Slimed-32K",
                    dataset_config={"name": "wikitext",
                                "config": "wikitext-103-raw-v1",
                                "split": "train",
                                "text_column": "text"},
                    target_vocab_size=32_000)

# vocabulary slimming and adjust embedding weight
vocSlim.prune()

# compare outputs between original and slimmed models with test text
vocSlim.check("What is the capital of France?")

# slimmed model and tokenizer are saved in save_path

Evaluation

To evaluate the performance after vocabulary slimming, we used lm-evaluation-harness to compare on several common datasets.

from transformers import AutoModelForCausalLM, AutoTokenizer
from lm_eval.models.huggingface import HFLM
from lm_eval import simple_evaluate

model_path = "./Qwen2.5-0.5B-Vocab-Slimed-32K"
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto").eval()
tokenizer = AutoTokenizer.from_pretrained(model_path)

lm = HFLM(pretrained=model, tokenizer=tokenizer, batch_size=32)

results = simple_evaluate(model=lm, tasks=["arc_challenge", "hellaswag", "piqa"])
results = {key: value for key, value in results.items() if key == "results"}
print("Accuracy:", results)

Model Size Reduction

Limitations

• Only Hugging Face models with BPE tokenizer are supported.
• May cause performance degradation due to reduced vocabulary size.
• Currently only supports causal language models

References

Papers

1. Efficient Vocabulary Reduction for Small Language Models

   • Authors: Yuta Nozaki, Dai Nakashima, Ryo Sato, Naoki Asaba, Shintaro Kawamura
   • Published in: Proceedings of the 31st International Conference on Computational Linguistics: Industry Track 2025
   • [Paper]

2. Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

   • Authors: Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, Ngai Wong
   • Published in: NeurIPS 2024
   • [Paper]

3. Efficient Multilingual Language Model Compression through Vocabulary Trimming

   • Authors: Asahi Ushio, Yi Zhou, Jose Camacho-Collados
   • Published in: EMNLP 2023
   • [Paper]

Related Projects

1. Introducing Minivoc: Faster and Memory-Efficient LLMs Through Vocabulary Reduction [WIP]
   • [Link]

Citation

If you use this software, please cite it as given below;

@software{vocab-slim,
author = {Andy1314Chen},
license = {Apache-2.0},
title = {{vocab-slim: A Tool for Hugging Face Model Vocabulary Slimming}}
url = {https://github.com/Andy1314Chen/vocab-slim}
}

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.