tml-renderers

tml-renderers is the default TML renderer for text and multimodal chat. It packages public message types, a native tokenizer, TMLv0 rendering and parsing, and media codecs for inference and training.

Installation

pip install tml-renderers

tml-renderers is a compiled extension that requires PyTorch 2.10 or newer.

Quickstart

from tml_renderers.chat import Message, MessageList, Text, Author, AuthorKind

message = Message(
    content=Text("Hello, world!"),
    author=Author(AuthorKind.User),
)
messages = MessageList([message])

OpenAI message compatibility

Partners can use OpenAI-shaped message dictionaries at the package boundary without rewriting their chat data into native classes:

from tml_renderers import chat, tokenizers, v0

openai_message_dicts = [
    {"role": "system", "content": "Answer concisely."},
    {"role": "user", "content": "What is 2 + 2?"},
]

openai_messages = chat.OpenAIMessage.from_oss_messages(openai_message_dicts)
round_tripped = chat.OpenAIMessage.to_oss_messages(openai_messages)

renderer = v0.Renderer(tokenizers.o200k_base_chat())
spans, parser = renderer.render_for_completion(openai_messages)

The compatibility surface covers role/content messages, multipart text and image_url content, assistant tool calls, and tool results. render_for_completion and render_for_sft also accept list[OpenAIMessage] directly. This is a message-format bridge; it does not implement the OpenAI network client or server API. Use MessageList.from_oss_messages(...) and MessageList.to_oss_messages() when you want the same conversion through a native MessageList.

Rendering and parsing (TMLv0)

Rendering converts structured chat messages into the exact token and media-span sequence consumed by the model. v0 uses the same TMLv0 grammar for both inference and training:

• render_for_completion(...) renders an inference prompt and returns a parser for turning generated model tokens back into messages.
• render_for_sft(...) renders supervised-training examples with aligned, per-token loss weights.

The parser performs the reverse operation, reconstructing structured messages from generated token spans:

from tml_renderers import chat, tokenizers, v0

tokenizer = tokenizers.o200k_base_chat()
renderer = v0.Renderer(tokenizer)

messages = [
    chat.Message(
        content=chat.Text("Hello"),
        author=chat.Author(chat.AuthorKind.User),
    )
]

spans, parser = renderer.render_for_completion(messages)
parsed = parser.parse(spans)

training_examples = renderer.render_for_sft(messages)

The tokenizer is a native o200k BPE tokenizer plus the fixed TMLv0 special tokens. No Hugging Face chat template or remote code is involved.

tokenizer.encode_ordinary("hello")
tokenizer.decode([24912])
tokenizer.encode_special("message_user")   # -> 200000
tokenizer.decode_special(200000)            # -> "message_user"

Full method surface:

method	maps	on invalid / partial UTF-8
encode_ordinary(text)	text → ids	— (special-token markup is encoded as literal text)
encode_special(name)	name → id	raises if name isn't a special token
decode(ids)	ids → text	lossy — invalid/partial bytes become U+FFFD, never raises (tiktoken parity)
decode_strict(ids)	ids → text	raises ValueError on invalid/partial UTF-8
decode_bytes(ids)	ids → bytes	raw bytes, no UTF-8 handling
decode_special(id)	id → name	None if id isn't a special token

Plus is_special_token(id), special_tokens(), and the eos_token / bos_token / all_special_tokens properties. decode is lossy so a truncated multi-byte tail (common when streaming) renders as � rather than raising — reach for decode_strict only when you want that raise.

render_for_completion accepts a MessageList, a list[Message], or a list[OpenAIMessage]. render_for_completion_with_effort(messages, effort) (with effort a float in [0, 1]) inserts a system message containing Thinking effort level: 0.9 before the first non-system message; the messages must not already contain a ThinkingEffort content. For SFT, render_for_sft returns TrainingExamples carrying per-token loss weights.

Media

tml-renderers includes the multimodal codec logic used by the default TMLv0 renderer:

• An ImagePointer is mapped to the model's black-padded image patch layout and rendered as an ImageAssetPointerTokenSpan with the expected image-token count.
• An AudioPointer is decoded, resampled to 16 kHz, DMel-encoded, and rendered as a DmelTokenSpan.

Image and audio messages use the same completion and training APIs as text:

from tml_renderers import chat, tokenizers, v0

renderer = v0.Renderer(tokenizers.o200k_base_chat())

image_msg = chat.Message(
    content=chat.ImagePointer(
        location="image.png",
        format=chat.ImageFormat.Png,
        width=512,
        height=512,
    ),
    author=chat.Author(chat.AuthorKind.User),
)

audio_msg = chat.Message(
    content=chat.AudioPointer(
        location="clip.wav",
        format=chat.AudioFormat.Wav,
        num_frames=48_000,
        sample_rate=16_000,
    ),
    author=chat.Author(chat.AuthorKind.User),
)

spans, _ = renderer.render_for_completion([image_msg, audio_msg])
dmel = next(s.span for s in spans if type(s.span).__name__ == "DmelTokenSpan")
print(dmel.dmel.shape)  # [num_audio_tokens, num_dmel_bins], uint8

Media locations are represented as pointers. Audio is read from local files; tml-renderers never fetches remote media.

Streaming

Parser.parse_token / parse_updates / flush_updates return incremental ParseUpdates, so you can render a completion as tokens arrive rather than only returning completed Messages. Each update is a StreamingMessageHeader, a StreamingContent delta, or a completed Message. See tests/test_v0_streaming.py for a worked example.

Token spans

Renderers work with token-span types you can construct directly:

from tml_renderers.chat import (
    EncodedTextTokenSpan,
    ImageAssetPointerTokenSpan,
    ImageFormat,
    PaddingTokenSpan,
    TokenSpan,
)

text = EncodedTextTokenSpan([101, 102])
image = ImageAssetPointerTokenSpan(
    location="image.png", format=ImageFormat.Png, width=512, height=512, num_tokens=256
)
padding = PaddingTokenSpan(4)

# Wrap in the oneof only where a call boundary asks for a TokenSpan.
wrapped = TokenSpan(text)

DmelTokenSpan is the encoded-audio span produced by the renderer's audio handler; dmel is a uint8 tensor of shape [num_audio_tokens, num_dmel_bins].

Using with Tinker

Install the latest Tinker SDK and Tinker Cookbook, then select the tml_v0 renderer in your Cookbook recipe. Cookbook handles model-specific configuration and converts rendered text, image, and audio inputs for sampling and SFT.

The lower-level tml_renderers.tinker helpers are also available when integrating directly with the Tinker SDK. Remote media is never fetched; provide local files or base64 image data.

Types

• Core: Message, MessageList, Author, AuthorKind, MessageChannel, MessageMetadata
• Content: Text, Thinking, ThinkingEffort, ImagePointer, AudioPointer, InvokeTool, StructuredToolCall, ToolError, ToolDeclareJson, ModelEndSampling
• Token spans: TokenSpan, EncodedTextTokenSpan, ImageAssetPointerTokenSpan, DmelTokenSpan, PaddingTokenSpan
• Formats: ImageFormat, AudioFormat
• Tool args: ToolArg, ToolMetadata. ToolArg.value is a JSON-encoded value string, matching the OpenAI function.arguments convention.
• Tool declarations: ToolSpecJson (one declared tool: name / description / type_name, plus parameters as a JSON-schema string; the constructor accepts parameters as a dict and serializes it), grouped into a ToolDeclareJson.

Examples

• examples/run_tml_v0.py — low-level render/parse of text, tools, and audio.
• examples/cookbook_tinker_demo.py — OpenAI-style dicts alongside native types.

License

Apache-2.0. See LICENSE.