TypeLLM: LLMs with type-safe generation

Homepage  •   Blog  •   Docs  •   Early Access  •   Contact 

Updates

• [2026/09/23] Added permutation averaging to improve the predictive distribution. See the blog post for details.
• [2026/09/22] Added depends_on dependency graphs with incremental prefix reuse.
• [2026/09/19] Added optional thinking mode with a per-field budget.
• [2026/09/18] Added constrained integer and number outputs.

Introduction

TypeLLM extends autoregressive LLMs with type-safe generation. Models can still think and generate freely when needed, while producing guaranteed typed outputs when structure matters. Define the output with a JSON Schema, and TypeLLM returns values your software can use directly.

TypeLLM was inspired by TypeSafe AI's Jev, while pursuing a different goal: extending autoregressive LLMs with typed outputs and richer interaction patterns without changing their architecture or weights. It keeps the models’ original generation and reasoning capabilities while adding multiple output types. TypeLLM is built on SGLang and works with existing open models.

Supported output types

String · Integer · Number · Boolean · Enum choice — See schemas and examples.

Features

1. No out-of-schema hallucinations — Choices stay within the allowed values.
2. Negligible output-token cost — Single-token categorical selection and bounded numeric decoding; optional thinking adds tokens.
3. Shared-prefix reuse — KV caching avoids reprocessing shared context.
4. Dependency-aware execution — Run decisions sequentially, batch independent fields, or declare depends_on to form a dependency graph.
5. Made for open autoregressive LLMs — Use compatible models you already serve with SGLang.
6. Supports thinking mode — Enable reasoning before the final constrained answer.
7. Permutation averaging — Reduce option-order bias on explicit enum questions with sampled or exhaustive orderings. See the docs.

Quick start

1. Serve a model with SGLang

Use SGLang to configure and serve a compatible autoregressive model on your local GPU server. This example uses Qwen3.8-27B; follow the Qwen3.8-27B SGLang deployment guide to start it with prefix caching enabled.

See Supported models for tested checkpoints and thinking behavior.

2. Run TypeLLM

pip install -U typellm

Point TypeLLMClient at the SGLang server's HTTP endpoint:

from typellm import TypeLLMClient

client = TypeLLMClient(
    "http://127.0.0.1:30000",
    model="Qwen/Qwen3.8-27B",
)

Example request:

result = client.generate(
    context="""
    Receipt from Hilton London
    Total: £324.50
    Employee travelled to London for a client meeting.
    """,
    questions={
        "merchant": {
            "type": "string",
            "instructions": "Return only the merchant name.",
        },
        "total": {
            "type": "number",
            "instructions": "Extract the total amount in GBP.",
        },
        "expense_type": {
            "type": "string",
            "enum": ["meal", "travel", "equipment"],
            "instructions": "What type of expense is this?",
        },
        "reimbursable": {
            "type": "boolean",
            "instructions": "Should this expense be reimbursed?",
        },
        "confidence": {
            "type": "number",
            "enum": [0.0, 0.25, 0.5, 0.75, 1.0],
            "instructions": "How confident are you?",
        },
    },
)

print(result)

Example return:

{
    "merchant": "Hilton London",
    "total": 324.5,
    "expense_type": "travel",
    "reimbursable": True,
    "confidence": 0.75,
}

Output types

TypeLLM supports finite decisions, numeric fields, and free text:

Enum choices support string, integer, and number types, with at most 24 values. The declared type validates the candidate values.

Enum and boolean fields select a single-token label. Numeric and text fields without enum generate values token by token.

A string without enum generates free text:

result = client.generate(
    context="The train ticket is for a client meeting.",
    questions={
        "summary": {"type": "string", "instructions": "Summarize in one sentence."},
    },
)

Use "maxLength": 100 to limit text to 100 Unicode characters. The separate text_max_tokens client setting defaults to 512 tokens per field. Incomplete, invalid, or over-length text raises SGLangError. minLength, pattern, and format are not supported.

For example, ask for a numeric answer without enumerating every possible value:

result = client.generate(
    context="Calculate the requested value accurately.",
    questions={
        "answer": {
            "type": "number",
            "instructions": "What is 17.5 multiplied by 4?",
        },
    },
)

print(result)
# {"answer": 70.0}

Numeric fields without enum accept optional minimum and maximum. The bounds are shown to the model and the generated value is validated against them; an out-of-range value raises ValueError instead of being returned. These fields generate plain decimal notation with at most 32 digits by default; set TypeLLMClient(numeric_max_digits=...) to adjust this limit.

Use instructions to tell the model what decision to make:

{
    "type": "string",
    "enum": ["billing", "technical", "account"],
    "instructions": "Which team should handle this ticket?",
}

If instructions is omitted, TypeLLM uses description or an instruction generated from the field name.

Thinking mode

Thinking is off by default. Enable it when constructing the client:

client = TypeLLMClient(
    "http://127.0.0.1:30000",
    model="Qwen/Qwen3.8-27B",
    thinking=True,
)
result = client.generate(context=context, questions=questions)

Set thinking_budget=2048 to cap reasoning per field; no explicit budget is set by default. If reasoning reaches its limit or ends early at a recognized turn terminator, TypeLLM closes a nonempty thinking block and proceeds to the typed answer. Empty unfinished reasoning and unrecognized stops raise an error.

Models with always-on thinking still reason with thinking=False; thinking_budget applies to them too. See Supported models.

Dependency-aware execution

The default execution="auto" selects batch execution unless a field declares depends_on.

Set the mode per request or on the client:

result = client.generate(
    context=context,
    questions=questions,
    execution="sequential",  # Or "batch" for independent fields
)

Batch execution shares the cached context across branches. Configure SGLang's --max-running-requests for the desired concurrency.

Dependency execution (depends_on)

Declare which earlier results a field needs. Forward references are allowed: fields do not need to be declared in execution order.

result = client.generate(
    context="The payments service is returning errors after a deployment.",
    questions={
        "system": {
            "type": "string",
            "enum": ["payments", "accounts", "search"],
            "instructions": "Which system is affected?",
        },
        "severity": {
            "type": "string",
            "enum": ["low", "medium", "high"],
            "instructions": "Assess severity for the affected system.",
            "depends_on": ["system"],
        },
        "deployment_related": {
            "type": "boolean",
            "instructions": "Is the incident related to a deployment?",
            "depends_on": ["system"],
        },
        "rollback": {
            "type": "boolean",
            "instructions": "Based on the incident assessments, should we roll back?",
            "depends_on": ["severity", "deployment_related"],
        },
    },
)

This runs system, then severity and deployment_related, then rollback. Each layer finishes before the next starts; unrelated branches remain separate.

• depends_on lists unique field names. Missing or empty lists mark independent roots.
• Fields receive their direct and transitive dependency results. Probability-returning dependencies contribute only their selected value.
• Unknown names, self-dependencies, duplicates, and cycles raise SchemaError.
• Any depends_on, including [], activates DAG execution in auto mode. Combining it with explicit batch or sequential raises SchemaError.
• Both questions and object-form schema.properties support dependencies. Returned keys follow field declaration order.

All fields execute. Dependencies do not change enums, substitute values into instructions, or conditionally skip fields. A failed layer stops subsequent layers.

Incremental prefix reuse along dependencies

TypeLLM extends parent prompts along dependency paths, retaining previous answers and reasoning for KV cache reuse. In a chain A → B → C, each step builds on the previous prefix; independent branches share their common prefix.

When a field depends on multiple parents, TypeLLM reuses one parent prefix and includes all dependency results. SGLang manages the cache; KV tensors from different branches are not merged.

Batch performance

A local run with Qwen3.8-27B NVFP4 on one NVIDIA RTX PRO 6000 Blackwell GPU used roughly 1,100 context tokens and 16 Boolean fields, with --max-running-requests 16.

Each branch reused 1,088 cached tokens. Results depend on the model, workload, and server configuration. Sequential fields see earlier answers; batch fields are independent, so the modes serve different workflows.

Probabilities and sampling

Set return_probabilities on individual enum or boolean fields:

result = client.generate(
    context=context,
    questions={
        "expense_type": {
            "type": "string",
            "enum": ["meal", "travel", "equipment"],
            "return_probabilities": True,
        },
    },
)

{
    "expense_type": {
        "value": "travel",
        "probabilities": {
            "meal": 0.04,
            "travel": 0.93,
            "equipment": 0.03,
        },
    }
}

Only opted-in fields return value and probabilities; other fields return plain values. The option is not supported on open Numeric or Text fields.

Per-question permutation averaging

Add permutations to an enum question to reduce option-order bias. TypeLLM averages the probabilities and keeps the same return format.

result = client.generate(
    context="A single roll of a fair die.",
    questions={"roll": {
        "type": "string",
        "enum": ["one", "two", "three", "four", "five", "six"],
        "instructions": "What number will come up on this roll?",
        "permutations": 8,
        "return_probabilities": True,
    }},
)

Use 8 for eight distinct orderings or "all" for every ordering (up to 720). Omit it or use 1 to keep the original behavior. Only explicit enum fields support this option.

Docs · Read the blog

Argmax is the default. To enable sampling:

client = TypeLLMClient(
    "http://127.0.0.1:30000",
    mode="sample",
    temperature=0.8,
    seed=42,
)

Sampling applies to finite candidates for Choice fields and to token generation for Numeric and Text fields. temperature controls sampling in each case.

For a one-off request, use the convenience function:

from typellm import run_schema

result = run_schema(
    context=context,
    questions=questions,
    base_url="http://127.0.0.1:30000",
    model="Qwen/Qwen3.8-27B",
)

Cost analysis

Enum and boolean fields use one output token each. Numeric, text, and optional thinking outputs use multiple tokens.

For a sequential workflow with D fields, C original context tokens, and roughly S new tokens per turn, input prefill counts are:

without prefix reuse: O(D*C + D^2*S)
with prefix reuse:    O(C + D*S)

For independent batch fields, the shared context is prefilled once, followed by each field's question. These counts describe input token positions, not GPU compute or latency: new tokens still attend to the cached prefix, and reuse depends on cache availability. Hosted billing depends on the provider's cached-input pricing.

Supported models

The following models have been tested with TypeLLM on a live SGLang GPU server.

Other sizes in the Qwen3.5 and Qwen3.8 families are expected to be compatible.

The MiniCPM5, Ling and Ring runs used an RTX PRO 6000 Blackwell and SGLang 0.5.19 on 2026-09-22.

Use the checkpoint ID as model=. If the server's tokenizer path is unavailable locally, set tokenizer= to its matching Hugging Face ID or local directory. The tokenizer must load from standard artifacts without custom model code.

Comparison with Jev-style models

© 2026 TypeLLM