tml-renderers

tml-renderers turns structured text, image, and audio chat into model-ready tokens and media for training and inference. It supports Thinking Machines Lab models, including Inkling and Inkling-Small, through TMLv0, and the supported third-party models served on Tinker through model-specific renderers.

Installation

Always use the latest release:

pip install --upgrade tml-renderers

tml-renderers is a compiled extension that requires PyTorch 2.10 or newer.

Quickstart

Create a short conversation with native chat objects and render it for inference or training:

from tml_renderers import chat, renderers

user = chat.Message(
    content=chat.Text("Hello, world!"),
    author=chat.Author(chat.AuthorKind.User),
)
assistant = chat.Message(
    content=chat.Text("Hello!"),
    author=chat.Author(chat.AuthorKind.Model),
)
native_messages = chat.MessageList([user, assistant])

renderer = renderers.Qwen3()
spans, parser = renderer.render_for_completion([user])
training_examples = renderer.render_for_sft(native_messages)

To continue a conversation, add the model response and new user or tool messages to the history and render again.

Supported models

Provider / family	Models	Renderer classes
thinkingmachines	Inkling, Inkling-Small	v0.Renderer (TMLv0)
Qwen	Qwen3 (including vision), Qwen3.5, Qwen3.8	Qwen3, Qwen35, Qwen38
DeepSeek	V3, R1, V3.1, V4, V4.1 Flash	DeepSeekV3, DeepSeekR1, DeepSeekV31, DeepSeekV4, DeepSeekV41
Kimi	K2, K2.5	KimiK2, KimiK25
GLM	5.2, 5.3, 5.3 Flash	Glm52, Glm53, Glm53Flash
GPT-OSS	GPT-OSS	GptOss
Nemotron	Nemotron 3	Nemotron3

OpenAI message compatibility

The same conversation can use OpenAI-style dictionaries:

Message	Native chat object	OpenAI-style dict
User	chat.Message(content=chat.Text("Hello, world!"), author=chat.Author(chat.AuthorKind.User))	{"role": "user", "content": "Hello, world!"}
Assistant	chat.Message(content=chat.Text("Hello!"), author=chat.Author(chat.AuthorKind.Model))	{"role": "assistant", "content": "Hello!"}

Convert the dictionaries before passing them to a renderer:

from tml_renderers import chat, renderers

openai_dicts = [
    {"role": "user", "content": "Hello, world!"},
    {"role": "assistant", "content": "Hello!"},
]
openai_messages = chat.OpenAIMessage.from_oss_messages(openai_dicts)
round_tripped = chat.OpenAIMessage.to_oss_messages(openai_messages)

renderer = renderers.Qwen3()
training_examples = renderer.render_for_sft(openai_messages)

The renderer also accepts openai_messages for completion. The conversion supports text, images, tool calls, and tool results; it handles message formats, not the OpenAI network API.

Rendering and parsing (TMLv0)

Rendering converts structured chat messages into the token and media-span sequence consumed by the model. v0 uses the same TMLv0 grammar for both inference and training:

• render_for_completion(...) renders an inference prompt and returns a parser for turning generated model tokens back into messages.
• render_for_sft(...) renders supervised-training examples with aligned, per-token loss weights.

The parser reconstructs structured messages from generated token spans:

from tml_renderers import chat, tokenizers, v0

tokenizer = tokenizers.O200kBaseChatTokenizer()
renderer = v0.Renderer(tokenizer)

messages = [
    chat.Message(
        content=chat.Text("Hello"),
        author=chat.Author(chat.AuthorKind.User),
    )
]

spans, parser = renderer.render_for_completion(messages)
parsed = parser.parse(spans)

training_examples = renderer.render_for_sft(messages)

The package includes native tokenizers for these renderers. It does not need Hugging Face chat templates or remote code.

tokenizer.encode_ordinary("hello")
tokenizer.decode([24912])
tokenizer.encode_special("message_user")   # -> 200000
tokenizer.decode_special(200000)            # -> "message_user"

qwen_tokenizer = tokenizers.Qwen3Tokenizer()
qwen_tokenizer.encode_special("im_start")   # -> 151644

Full method surface:

method	maps	on invalid / partial UTF-8
encode_ordinary(text)	text → ids	— (special-token markup is encoded as literal text)
encode_special(name)	name → id	raises if name isn't a special token
decode(ids)	ids → text	lossy — invalid/partial bytes become U+FFFD, never raises (tiktoken parity)
decode_strict(ids)	ids → text	raises ValueError on invalid/partial UTF-8
decode_bytes(ids)	ids → bytes	raw bytes, no UTF-8 handling
decode_special(id)	id → name	None if id isn't a special token

The tokenizer also provides is_special_token(id), special_tokens(), and eos_token, bos_token, and all_special_tokens. decode is lossy: a truncated multibyte tail renders as � rather than raising. Use decode_strict when invalid UTF-8 must raise an error.

render_for_completion accepts a MessageList, a list[Message], or a list[OpenAIMessage]. For TMLv0, render_for_completion_with_effort(messages, effort) accepts an effort float in [0, 1] and inserts a system message before the first non-system message; input messages must not already contain ThinkingEffort. For SFT, render_for_sft returns TrainingExamples with per-token loss weights.

Multimodal inputs

tml-renderers includes multimodal codecs for the TMLv0 renderer:

• An ImagePointer is rendered as an ImageAssetPointerTokenSpan with the expected image-token count.
• An AudioPointer is rendered as a DmelTokenSpan.

Image and audio messages use the same completion and training APIs as text:

from tml_renderers import chat, tokenizers, v0

renderer = v0.Renderer(tokenizers.O200kBaseChatTokenizer())

image_msg = chat.Message(
    content=chat.ImagePointer(
        location="image.png",
        format=chat.ImageFormat.Png,
        width=512,
        height=512,
    ),
    author=chat.Author(chat.AuthorKind.User),
)

audio_msg = chat.Message(
    content=chat.AudioPointer(
        location="clip.wav",
        format=chat.AudioFormat.Wav,
        num_frames=48_000,
        sample_rate=16_000,
    ),
    author=chat.Author(chat.AuthorKind.User),
)

spans, _ = renderer.render_for_completion([image_msg, audio_msg])
dmel = next(s.span for s in spans if type(s.span).__name__ == "DmelTokenSpan")
print(dmel.dmel.shape)  # [num_audio_tokens, num_dmel_bins], uint8

Media locations are represented as pointers. Audio is read from local files; tml-renderers never fetches remote media.

Token spans

Renderers work with token-span types you can construct directly:

from tml_renderers.chat import (
    EncodedTextTokenSpan,
    ImageAssetPointerTokenSpan,
    ImageFormat,
    PaddingTokenSpan,
    TokenSpan,
)

text = EncodedTextTokenSpan([101, 102])
image = ImageAssetPointerTokenSpan(
    location="image.png", format=ImageFormat.Png, width=512, height=512, num_tokens=256
)
padding = PaddingTokenSpan(4)

# Wrap in the oneof only where a call boundary asks for a TokenSpan.
wrapped = TokenSpan(text)

DmelTokenSpan is the encoded-audio span produced by the renderer. Its dmel field is a uint8 tensor of shape [num_audio_tokens, num_dmel_bins].

Using with Tinker

In Tinker Cookbook, use the recommended renderer for your model. Select tml_v0 for Inkling and Inkling-Small; supported third-party models use their model-specific renderers. Cookbook converts rendered text, image, and audio inputs for sampling and SFT.

The lower-level tml_renderers.tinker helpers are also available for direct Tinker SDK integration. Provide local media files or base64 image data.

Types

• Core: Message, MessageList, Author, AuthorKind, MessageChannel, MessageMetadata
• Content: Text, Thinking, ThinkingEffort, ImagePointer, AudioPointer, InvokeTool, StructuredToolCall, ToolError, ToolDeclareJson, ModelEndSampling
• Token spans: TokenSpan, EncodedTextTokenSpan, ImageAssetPointerTokenSpan, DmelTokenSpan, PaddingTokenSpan
• Formats: ImageFormat, AudioFormat
• Tool args: ToolArg, ToolMetadata. ToolArg.value is a JSON-encoded string, matching the OpenAI function.arguments convention.
• Tool declarations: ToolSpecJson (one declared tool with name, description, type_name, and JSON-schema parameters), grouped into ToolDeclareJson. The constructor accepts a dict for parameters and serializes it.

Examples

• examples/run_tml_v0.py — low-level rendering and parsing of text and audio.
• examples/cookbook_tinker_demo.py — native and OpenAI-style messages, including tool calls and results. See the public tool-call example.

License

Apache-2.0. The wheel includes LICENSE, NOTICE, and THIRD_PARTY_LICENSES.md.