Baseten Tokenizer

High-performance, Rust-backed BPE tokenization for inference. The Python package is named basetenkenizer.

Install

pip install basetenkenizer

Loading by Hugging Face model ID downloads tokenizer.json on first use and then uses the local Hugging Face cache. Private or gated models use the usual HF_TOKEN environment variable.

Encode text

from basetenkenizer import Tokenizer

tokenizer = Tokenizer.from_model("deepseek-ai/DeepSeek-V3.2")
encoding = tokenizer.encode(
    "A very long prompt that is now much faster.",
    add_special_tokens=False,
)

print(encoding.ids)
print(tokenizer.decode(encoding.ids))

Tokenizer.from_file("tokenizer.json") loads a local tokenizer. Encoding objects expose ids, attention_mask, type_ids, and special_tokens_mask; selected fields can be moved into NumPy arrays with encoding.into_numpy(...).

Kimi K2.7 Code

Load the published tokenizer directly from its model repository:

from basetenkenizer import Tokenizer

tokenizer = Tokenizer.from_model("moonshotai/Kimi-K2.7-Code")
ids = tokenizer.encode("def hello():\n    return 'world'").ids

For a chat prompt, preserve the boundary between template control tokens and untrusted message content with encode_segments:

user_message = "Write a Python HTTP server."

segments = [
    ("<|im_user|>user<|im_middle|>", True),
    (user_message, False),
    ("<|im_end|>", True),
    ("<|im_assistant|>assistant<|im_middle|><think>", True),
]

encoding = tokenizer.encode_segments(segments)
input_ids = encoding.ids

This is the minimal Kimi K2.7 Code user/assistant shape. Applications using system messages, tools, images, or existing assistant messages should render the complete official model template and retain the same control-text versus message-text boundaries.

Kimi K3

Kimi K3 uses XTML and a Python chat renderer rather than a Jinja chat template. Raw text can still be encoded normally:

from basetenkenizer import Tokenizer

tokenizer = Tokenizer.from_model("moonshotai/Kimi-K3")
ids = tokenizer.encode("Hello from Kimi K3").ids

For rendered chat, pass the official Kimi K3 renderer's EncodeSegment objects to encode_segments as (segment.text, segment.allow_special). A minimal user message followed by an assistant-generation prefix looks like this:

user_message = "Explain speculative decoding."

segments = [
    ("<|open|>", True),
    ("message", False),
    (" role", False),
    ('="', False),
    ("user", False),
    ('"', False),
    ("<|sep|>", True),
    (user_message, False),
    ("<|close|>", True),
    ("message", False),
    ("<|sep|>", True),
    ("<|end_of_msg|>", True),
    ("<|open|>", True),
    ("message", False),
    (" role", False),
    ('="', False),
    ("assistant", False),
    ('"', False),
    ("<|sep|>", True),
    ("<|open|>", True),
    ("response", False),
    ("<|sep|>", True),
]

encoding = tokenizer.encode_segments(segments)
input_ids = encoding.ids

Use Kimi K3's official build_chat_segments renderer for production conversations involving tools, images, reasoning content, response schemas, or multiple message types. Do not concatenate its segments before encoding: doing so loses the special-token safety boundary.

Why `encode_segments`?

Each segment is a (text, allow_special) pair:

allow_special=True recognizes tokenizer control tokens emitted by a trusted chat renderer.
allow_special=False treats control-token-looking strings in user, tool, or attribute content as ordinary text.

tiktoken_safe=True is the default. It reproduces the chunk boundaries used by legacy tiktoken tokenizers, including on very long inputs, so token IDs stay compatible. Set it to False only when exact tiktoken parity is not required and whole-segment BPE encoding is intentional.

Post-processing, truncation, and padding are applied after all segment IDs have been joined.

Use with Transformers

Call patch_transformers before loading a tokenizer:

import basetenkenizer

basetenkenizer.patch_transformers()

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-120b")
tokens = tokenizer("Hello, world!")

Transformers v4 and v5 are supported. Pass patch_transformers(apply_chat_template=True) to also use the native renderer for supported render-only apply_chat_template(..., tokenize=False) calls; unsupported templates automatically fall back to Transformers.

Baseten Tokenizer is focused on inference and does not implement every Hugging Face Tokenizers training or alignment feature.

License

MIT

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

basetenkenizer-0.2.8-cp314-cp314t-musllinux_1_2_x86_64.whl (4.8 MB view details)

Uploaded Jul 27, 2026 CPython 3.14tmusllinux: musl 1.2+ x86-64

basetenkenizer-0.2.8-cp314-cp314t-musllinux_1_2_i686.whl (4.7 MB view details)

Uploaded Jul 27, 2026 CPython 3.14tmusllinux: musl 1.2+ i686

basetenkenizer-0.2.8-cp314-cp314t-musllinux_1_2_armv7l.whl (4.4 MB view details)

Uploaded Jul 27, 2026 CPython 3.14tmusllinux: musl 1.2+ ARMv7l

basetenkenizer-0.2.8-cp314-cp314t-musllinux_1_2_aarch64.whl (4.5 MB view details)

Uploaded Jul 27, 2026 CPython 3.14tmusllinux: musl 1.2+ ARM64

basetenkenizer-0.2.8-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded Jul 27, 2026 CPython 3.14tmanylinux: glibc 2.17+ x86-64

basetenkenizer-0.2.8-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (5.0 MB view details)

Uploaded Jul 27, 2026 CPython 3.14tmanylinux: glibc 2.17+ ppc64le

basetenkenizer-0.2.8-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl (4.7 MB view details)

Uploaded Jul 27, 2026 CPython 3.14tmanylinux: glibc 2.17+ i686

basetenkenizer-0.2.8-cp314-cp314t-macosx_11_0_arm64.whl (4.0 MB view details)

Uploaded Jul 27, 2026 CPython 3.14tmacOS 11.0+ ARM64

basetenkenizer-0.2.8-cp314-cp314t-macosx_10_12_x86_64.whl (4.1 MB view details)

Uploaded Jul 27, 2026 CPython 3.14tmacOS 10.12+ x86-64

basetenkenizer-0.2.8-cp310-abi3-win_amd64.whl (3.7 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+Windows x86-64

basetenkenizer-0.2.8-cp310-abi3-musllinux_1_2_x86_64.whl (4.8 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+musllinux: musl 1.2+ x86-64

basetenkenizer-0.2.8-cp310-abi3-musllinux_1_2_i686.whl (4.7 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+musllinux: musl 1.2+ i686

basetenkenizer-0.2.8-cp310-abi3-musllinux_1_2_armv7l.whl (4.4 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+musllinux: musl 1.2+ ARMv7l

basetenkenizer-0.2.8-cp310-abi3-musllinux_1_2_aarch64.whl (4.5 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+musllinux: musl 1.2+ ARM64

basetenkenizer-0.2.8-cp310-abi3-manylinux_2_28_ppc64le.whl (5.0 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+manylinux: glibc 2.28+ ppc64le

basetenkenizer-0.2.8-cp310-abi3-manylinux_2_28_armv7l.whl (4.2 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+manylinux: glibc 2.28+ ARMv7l

basetenkenizer-0.2.8-cp310-abi3-manylinux_2_28_aarch64.whl (4.5 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+manylinux: glibc 2.28+ ARM64

basetenkenizer-0.2.8-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+manylinux: glibc 2.17+ x86-64

basetenkenizer-0.2.8-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (4.7 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+manylinux: glibc 2.17+ i686

basetenkenizer-0.2.8-cp310-abi3-macosx_11_0_arm64.whl (4.0 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+macOS 11.0+ ARM64

basetenkenizer-0.2.8-cp310-abi3-macosx_10_12_x86_64.whl (4.1 MB view details)

Uploaded Jul 27, 2026 CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file basetenkenizer-0.2.8-cp314-cp314t-musllinux_1_2_x86_64.whl.

File metadata

Download URL: basetenkenizer-0.2.8-cp314-cp314t-musllinux_1_2_x86_64.whl
Upload date: Jul 27, 2026
Size: 4.8 MB
Tags: CPython 3.14t, musllinux: musl 1.2+ x86-64
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.14.1

File hashes

Hashes for basetenkenizer-0.2.8-cp314-cp314t-musllinux_1_2_x86_64.whl
Algorithm	Hash digest
SHA256	`f35e9d1ea1b433fcfa804559e91ff7b9a7b3d163bda7351490fec20651dd7634`
MD5	`968374e692f59f6704f0c3143972198a`
BLAKE2b-256	`9b27e96e027116a14657e69ade41b0fc32281c91b79241d4f5789ec54b0c5c4b`

Algorithm	Hash digest
SHA256	`9cafa215b511e394685a834e38cbe28fbca6ec467c08dd5ff602ce91ce78f089`
MD5	`e208368f016c6ae4049a963907545a56`
BLAKE2b-256	`e34400fd0190c55fc9b5302f45ba47e2c77a5d61b8d44fdd5b9b515563a601e8`

Algorithm	Hash digest
SHA256	`10947213ed80667ef48f4a74142951d617ea7e6be7a6e39fb2d4bbb8e2fe535f`
MD5	`f46cd4751c3ce0425e8b2642b7a23462`
BLAKE2b-256	`683152c7524949474b53a87c50e23601452d9e27135223d6ffa50529681cde31`

Algorithm	Hash digest
SHA256	`48c2f94f797ee1a9a39b9d87193d40db4bafb24bed9bf4d5ee2e812cf9016615`
MD5	`0217abf4544c8f5c92255c91725eb697`
BLAKE2b-256	`055f1806aa465cf2a6e5f8ba376fc792eb13d4f24a5d4d639be4cc8c8823c609`

Algorithm	Hash digest
SHA256	`3ba931ef9d73798a130df5c2694a6687341a19dce03a585607aab052cb383c63`
MD5	`98786fd33727564333b44f6175504813`
BLAKE2b-256	`650af0b7c6db0c31f902b6af6f87de87ae60477c1205ea4bf7e7af235b8e6438`

Algorithm	Hash digest
SHA256	`6b4f029d56a2ee3a8b91f9b2b24ead7df20feba3a165da26ef23213bb3c47c67`
MD5	`4f48751296dc5fc3d078d14ce96a8520`
BLAKE2b-256	`630192a3aebda215ef2e143b04ff9000dccf06371efc02da3a30ce11cc6df984`

Algorithm	Hash digest
SHA256	`fe454efb1f2c0a78044afb6945a6ba3c91d30f4f8c0ab41182886757985bc385`
MD5	`b0e6988ed6d2b0c838173b5f2d1884fc`
BLAKE2b-256	`3e042ef90ba3f034a63385f40e93e562f4990960ef863ff6944ed643371cc17f`

Algorithm	Hash digest
SHA256	`31044c8114f9cc406fa7bb1f6511b38f308a4c915528d9e4f1c3edc3b5bc2521`
MD5	`d7eda4ee1e4abfdb0b75a6115a427fb5`
BLAKE2b-256	`6a18e9eb596f4433b0263f2c2e2ca795af05eefe806108783604d46f81425b96`

Algorithm	Hash digest
SHA256	`c559b458a7fb14e299f1a3b1c07b35ac9fe75e3f8ccfa6459866091a52f8ee52`
MD5	`75efba026568cef5cef6a997ee70e433`
BLAKE2b-256	`fad908b018b8a9f711a2252215fa8c6dbc3bb0d291dc013c36f145fc812d0c8d`

Algorithm	Hash digest
SHA256	`4248b8717a9d6abd8e92948458aa59d73faebab8a169eb46ca15eb518834c34f`
MD5	`db9bf2a658de8d80dcaaae2554e15d12`
BLAKE2b-256	`6f1758b3bf0af8a2217cb0b99e86edaf6d557fe35d422c6d599d7dc5f3de22cf`

Algorithm	Hash digest
SHA256	`6864ce7961b61432cbe2d6df474822dc741c93effeaa28ba52779f21b864164b`
MD5	`dad5bee8387252095b531e60eb3ca512`
BLAKE2b-256	`a5bd86ba1d15b2655bb24e11b6d7e5f78164dd43e7ba3b7573e198dd7f56856c`

Algorithm	Hash digest
SHA256	`c5f302c1830697a94a6862aab656d1f910cdbea939d699fc96ff38aef12e48a5`
MD5	`a5750b21a77dfa1ad780da301e2d7359`
BLAKE2b-256	`455aac6b796f7144cd27649ee5e640a71a8fae3354d6ac68183ee1abb95f3262`

Algorithm	Hash digest
SHA256	`8254c61321c55e9ddf5660f533601806c7d56f7ba0c6f376ac31b7be1222dea2`
MD5	`654e66d319c69f20bbf15ec7447a7c55`
BLAKE2b-256	`dd1b51706c7932d18cc03db8dd13c5124fe9b0c1734b9cb63ae3d6bf956aedd8`

Algorithm	Hash digest
SHA256	`17476e23e95550eee7d2f0f3e3b8ce022e67094a90bca8dc47f100184b9acbc8`
MD5	`71987f0dd84e23136aa40a412baedb44`
BLAKE2b-256	`0d5dd9bf9fb52e7cad9b01d635092050cf9bf10ff5a6810b5fbdb968afcffbf7`

Algorithm	Hash digest
SHA256	`5042c8e334d0d457a0800248bc67a41278af882a365ae47638f3645861068533`
MD5	`d397323c7f5bd793d21baeec87af3105`
BLAKE2b-256	`e30e0f4591f6af4858ceb23c4131a25de34012dcd6c656c98a6721fa72e319f3`

Algorithm	Hash digest
SHA256	`9561f4161c4e9f83693b5bf478bd3ef45e9db40bf10a7cc928d253080444089f`
MD5	`4d71cf3aaedf5d9ee8f616b360626bbf`
BLAKE2b-256	`8b70984520f00ed809ea8e98a1fd8cce6b4c21431774981c609d04f55e6b593f`

Algorithm	Hash digest
SHA256	`6a019f3bf96ba6e2c93b500496ea4123e692cf6112cfe0971601b23963d25461`
MD5	`93a07559aac2f9ea76a0aaba8f215a57`
BLAKE2b-256	`cfc80baf69219caa8e60945cee6ca4025c7b20c9e257c9d465f33e2422b3a9ff`

Algorithm	Hash digest
SHA256	`7bbe01584a65baf9d01b90d62b2513b4d4b6831bddca447cdbe6e24da78ae24c`
MD5	`df11881a25c16d9f4f93c425fcb6fc24`
BLAKE2b-256	`a9ff5e36b4b0eb1a5bd9160e14f7ab7eb0e5a02a505743dd328febe1a545c87f`

Algorithm	Hash digest
SHA256	`05f1ac7aa8f6edd326457fd305cda2a78c6180d8b07a52e423dde19a67196887`
MD5	`5bc3231c9113ff76f455423c03be45b9`
BLAKE2b-256	`6abfcbc57612c7b19fe6842035cfaf7cd7a0beab1d22a95f956656c6f199350e`

Algorithm	Hash digest
SHA256	`52d677bf7bf61fb25d2e71f090014ef8c80ab050264c0db8c66928a64fef287a`
MD5	`612d8ba11e15270702686374ba3bfaae`
BLAKE2b-256	`418fe6671af8d53adb902ce7896b777af77f2b56c7322c9359eaafee5048e944`

Algorithm	Hash digest
SHA256	`b3c7a23835f9639e8fc776ca9ec2c2480ba016db38b9dab5dbadd1af735a22ca`
MD5	`74ac8beb0ed3d0189a0ea915ee1fa6f3`
BLAKE2b-256	`1e015b1313cd552e18be10e470162cd59de75c5bf0896c882205409abb925f68`

basetenkenizer 0.2.8

Baseten Tokenizer

Install

Encode text

Kimi K2.7 Code

Kimi K3

Why encode_segments?

Use with Transformers

License

Project links

Key dates

1 maintainer

Credits

Requires

Download files

Source Distributions

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

Why `encode_segments`?