Qwen Tokenizer
Introduction
Qwen Tokenizer is a lightweight tokenization library that does not require the transformers package. It uses tiktoken as its only runtime dependency.
Version 0.3.0 adds the tokenizer shared by the official Qwen/Qwen3.5-* and Qwen/Qwen3.6-* models, including Qwen/Qwen3.6-27B. Earlier Qwen models continue to use the legacy tokenizer, so selecting a Qwen3.5 or Qwen3.6 model does not change tokenization for Qwen2.5 applications.
Installation
To install Qwen Tokenizer, use the following command:
pip install qwen_tokenizer
Basic Usage
Below is a simple example demonstrating how to use the Qwen3.6 tokenizer:
from qwen_tokenizer import get_tokenizer
qwen_tokenizer = get_tokenizer("qwen3.6-27b")
text = "Hello! 毕老师！1 + 1 = 2 ĠÑĤвÑĬÑĢ"
result = qwen_tokenizer.encode(text)
print(result)
Output
[9419, 0, 220, 97169, 97621, 6115, 16, 478, 220, 16, 283, 220, 17, 9551, 254, 68933, 128, 97, 91914, 28495, 68933, 128, 105, 68933, 128, 95]
Hugging Face-style model IDs are also accepted:
qwen_tokenizer = get_tokenizer("Qwen/Qwen3.6-27B")
Qwen3.5 and Qwen3.6 use a 248,044-token base vocabulary plus 33 added tokens. For example, <|im_start|>, <think>, and </think> map to IDs 248045, 248068, and 248069 respectively.
License
This project is licensed under the MIT License.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file qwen_tokenizer-0.3.0.tar.gz.
File metadata
- Download URL: qwen_tokenizer-0.3.0.tar.gz
- Upload date:
- Size: 3.2 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
5f6ae00b1297dd857c502732b30ba511481bf41cfe2a4ee021431fbb89029c8f
|
|
| MD5 |
1bfb09e2741aebc047c93844a923aeda
|
|
| BLAKE2b-256 |
bc036e0507f5a0b439d19849140184b4b597f0beb784905c75897fe4e1895f46
|
Provenance
The following attestation bundles were made for qwen_tokenizer-0.3.0.tar.gz:
Publisher:
release.yml on AndersonBY/qwen-tokenizer
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
qwen_tokenizer-0.3.0.tar.gz -
Subject digest:
5f6ae00b1297dd857c502732b30ba511481bf41cfe2a4ee021431fbb89029c8f - Sigstore transparency entry: 2142703157
- Sigstore integration time:
-
Permalink:
AndersonBY/qwen-tokenizer@3ab5b2b338eeba86768ee81bf6224d2b71e79992 -
Branch / Tag:
refs/tags/v0.3.0 - Owner: https://github.com/AndersonBY
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@3ab5b2b338eeba86768ee81bf6224d2b71e79992 -
Trigger Event:
push
-
Statement type:
File details
Details for the file qwen_tokenizer-0.3.0-py3-none-any.whl.
File metadata
- Download URL: qwen_tokenizer-0.3.0-py3-none-any.whl
- Upload date:
- Size: 3.2 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
cfcd0f49459aeba14f9eab32cd4eff487c4871de41ecfd5d6f783db343afca90
|
|
| MD5 |
6476cc4d5cfb19e0bf7d5d4d08fbab61
|
|
| BLAKE2b-256 |
57a18b18cdba867a2fdf686463eeef134a0003838322717c23bacaa9b2459a37
|
Provenance
The following attestation bundles were made for qwen_tokenizer-0.3.0-py3-none-any.whl:
Publisher:
release.yml on AndersonBY/qwen-tokenizer
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
qwen_tokenizer-0.3.0-py3-none-any.whl -
Subject digest:
cfcd0f49459aeba14f9eab32cd4eff487c4871de41ecfd5d6f783db343afca90 - Sigstore transparency entry: 2142703164
- Sigstore integration time:
-
Permalink:
AndersonBY/qwen-tokenizer@3ab5b2b338eeba86768ee81bf6224d2b71e79992 -
Branch / Tag:
refs/tags/v0.3.0 - Owner: https://github.com/AndersonBY
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@3ab5b2b338eeba86768ee81bf6224d2b71e79992 -
Trigger Event:
push
-
Statement type:
