basetenkenizer 0.2.6

High-performance tokenization for inference

Baseten Tokenizer

High-performance tokenization for inference, backed by Rust.

Install

pip install basetenkenizer

Standalone

from basetenkenizer import Tokenizer

tokenizer = Tokenizer.from_model("deepseek-ai/DeepSeek-V3.2")
encoding = tokenizer.encode(
    "A very long prompt that is now much faster.",
    add_special_tokens=False,
)
print(encoding.ids)

Kimi K3 rendered prompts

For a rendered prompt, encode_segments keeps template control text separate from user content. Each segment is (text, allow_special): use True for template-generated text where added/control tokens should be recognized, and False for user or tool content where those strings must remain ordinary text. The default tiktoken_safe=True preserves the chunking behavior used by the Kimi tokenizer.

segments = [
    ("<|im_start|>user\n", True),
    (user_message, False),
    ("<|im_end|>\n<|im_start|>assistant\n", True),
]
encoding = tokenizer.encode_segments(segments, add_special_tokens=False)
token_ids = encoding.ids

Transformers

import basetenkenizer

basetenkenizer.patch_transformers()

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-120b")
tokens = tokenizer("Hello, world!")

Baseten Tokenizer is focused on inference and does not implement every feature of Hugging Face Tokenizers.

Learn more about fast inference at baseten.co.

License

MIT