Qwen Tokenizer

Introduction

Qwen Tokenizer is a lightweight tokenization library that does not require the transformers package. It uses tiktoken as its only runtime dependency.

Version 0.3.0 adds the tokenizer shared by the official Qwen/Qwen3.5-* and Qwen/Qwen3.6-* models, including Qwen/Qwen3.6-27B. Earlier Qwen models continue to use the legacy tokenizer, so selecting a Qwen3.5 or Qwen3.6 model does not change tokenization for Qwen2.5 applications.

Installation

To install Qwen Tokenizer, use the following command:

pip install qwen_tokenizer

Basic Usage

Below is a simple example demonstrating how to use the Qwen3.6 tokenizer:

from qwen_tokenizer import get_tokenizer


qwen_tokenizer = get_tokenizer("qwen3.6-27b")
text = "Hello! 毕老师！1 + 1 = 2 ĠÑĤвÑĬÑĢ"

result = qwen_tokenizer.encode(text)
print(result)

Output

[9419, 0, 220, 97169, 97621, 6115, 16, 478, 220, 16, 283, 220, 17, 9551, 254, 68933, 128, 97, 91914, 28495, 68933, 128, 105, 68933, 128, 95]

Hugging Face-style model IDs are also accepted:

qwen_tokenizer = get_tokenizer("Qwen/Qwen3.6-27B")

Qwen3.5 and Qwen3.6 use a 248,044-token base vocabulary plus 33 added tokens. For example, <|im_start|>, <think>, and </think> map to IDs 248045, 248068, and 248069 respectively.

License

This project is licensed under the MIT License.