Highlights

• Lossless speculative decoding — DFlash2 or MTP drafting with batch-invariant kernels: greedy output with speculation on is token-identical to speculation off.
• Prefix cache for hybrid models — exact checkpoints for attention + GatedDeltaNet models, in RAM and on SSD, surviving restarts; optional extra storage tiers.
• The full API on the fast path — tools, JSON schema, stop, logprobs, reasoning, cancellation, across OpenAI Chat / Responses, Anthropic Messages and Ollama.
• Native coding-agent support — Claude Code, Codex and opencode work through their own APIs, including server-side web search / fetch and MCP.
• Lossless by default — anything that can change output is an explicit setting.
• Local and private — no telemetry; diagnostics never contain prompts.

Quickstart

Apple Silicon, macOS 14+, Python 3.13+ and uv.

uv tool install "yunshu[vision]"
yunshu doctor                                   # checks this Mac and says how to fix problems
yunshu pull mlx-community/Qwen3.5-9B-MLX-4bit
yunshu serve -m mlx-community/Qwen3.5-9B-MLX-4bit

The server listens on http://127.0.0.1:8000. Any OpenAI or Anthropic client works unchanged:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")  # any key works
r = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Explain MLX in one sentence."}],
)
print(r.choices[0].message.content)

from anthropic import Anthropic

client = Anthropic(base_url="http://127.0.0.1:8000", api_key="local")
msg = client.messages.create(
    model="local", max_tokens=512,
    messages=[{"role": "user", "content": "Explain MLX in one sentence."}],
)
print(msg.content[0].text)

Models live in ~/.yunshu/models (yunshu config set models_dir PATH moves them); serve -m org/name also finds the Hugging Face cache and downloads only if needed. yunshu service install -m <model> runs the server at login.

Qwen3.8-27B

yunshu pull Jundot/Qwen3.8-27B-oQ4e-mtp
yunshu pull incoai/Qwen3.8-27B-DFlash2          # optional drafter, picked up automatically
yunshu serve -m Jundot/Qwen3.8-27B-oQ4e-mtp

With the drafter installed the startup log says Speculative decoding: dflash; without it the model's own MTP head drafts. yunshu doctor -m <model> reports the selected path and whether the model fits this Mac's memory.

From source

git clone https://github.com/YuhuanStudio/Yunshu.git && cd Yunshu
uv sync --extra vision
uv run yunshu serve -m <model>

How it works

 OpenAI / Anthropic / Ollama clients ──► FastAPI gateway (one process)
                                           │  request validation, tool/reasoning parsing,
                                           │  server-side tools (web search / fetch / MCP)
                                           ▼
                                  engine (one MLX thread)
          ┌────────────────────────────────┴───────────────────────────────┐
   VLM batch runner (every mlx-vlm model)                 text fast path (mlx-lm models)
   shared continuous batch, per-row sampling              single-request generate_step
   speculative lane: DFlash2 / MTP / prompt-copy
   prefix cache: RAM ─► SSD ─► optional storage tiers

All GPU work runs on a single MLX thread, so requests never fight over the GPU. Each response returns as soon as its own generation ends.

Speculative decoding

For Qwen3.5-family models a single request decodes in a speculative lane:

• DFlash2 — a separate block drafter proposes several tokens per round; used automatically when a matching drafter is installed.
• MTP — the checkpoint's own multi-token-prediction head; the fallback when no drafter is present.
• Prompt-copy drafting — when the output starts repeating text from the prompt (code edits, quoting tool results, multi-turn agents), the lane proposes the continuation of that earlier text and verifies it in the same pass. It is on by default (YUNSHU_SPEC_COPY_ROWS, 0 turns it off).

Every decode and verify matmul goes through batch-invariant kernels: a token's arithmetic is the same whether it is verified alone or among other rows. That is what makes greedy output identical with speculation on and off, and sampled output exact under position-keyed sampling. Choose the drafter with YUNSHU_VLM_DRAFT (mtp, off, or a drafter path).

Prefix cache (APC)

Qwen3.5-family models mix attention layers with recurrent GatedDeltaNet layers. A recurrent state cannot be cut back to an earlier token the way a KV cache can, so ordinary prefix caching does not work. Yunshu stores exact checkpoints (KV plus recurrent state) at prefix boundaries, keyed by the text tokens and by image pixels / audio features, so a cache hit produces exactly what a cold prefill would.

Repeated or edited long prompts, multi-turn chats and agent loops restore from the nearest checkpoint instead of prefilling again. When a conversation grows, older checkpoints of the same conversation are replaced rather than piling up. yunshu cache status and yunshu cache gc show and clean the SSD caches.

Structured output

JSON schema, JSON object, regex and grammar constraints are enforced during decoding (llguidance by default), including tool-call arguments. Unsupported schema constructs are rejected with an error instead of being silently ignored.

API compatibility

Parameters covered on chat: tools / tool_choice / parallel_tool_calls, response_format (json_object, strict json_schema), stop, logprobs / top_logprobs (also streamed), n, seed, penalties, logit_bias, reasoning_effort (reasoning returned separately), streaming with usage, and cached-token counts in usage. Errors use each API's own error shape. Extensions are namespaced (x_yunshu, X-Yunshu-*), so the official SDKs ignore them. The full matrix, with how each row was verified, is in API surface.

Coding agents

yunshu launch claude      # or: codex, opencode

yunshu launch writes the client configuration (base URL, model, context window and output limits, reasoning effort) and starts the agent. For Claude Code it also installs a status line showing live prefill progress, decode speed and cache hits.

• Claude Code — Messages API with streaming, thinking, count_tokens for /context, model discovery, and its WebSearch tool, which Yunshu runs server-side.
• Codex — Responses API with reasoning items, function calls, local compaction and web_search.
• opencode — Chat Completions with tools and usage.

Server-side web search uses a configurable backend (for example SearXNG); MCP servers named in a request are connected by the gateway. What each agent calls and how it was checked is in Agent compatibility.

Performance

Qwen3.8-27B (oQ4e) on an M5 Max, 128 GB, single request, greedy. Methods, raw results and the full comparison tables are in docs/BENCHMARKS.md.

TensorFold is still faster at single-request decode and at 32K follow-up turns; closing those gaps is the main ongoing work. Structured output keeps speculative decoding (23 tok/s without it). Speculation never changes Yunshu's greedy output. Accuracy against the stock MLX path is checked at three levels (logit alignment, greedy divergence, paired downstream evals) in Accuracy.

Models

/v1/models returns each model's card: context length, output limit, and which inputs and features it actually supports.

Other capabilities

Command line

Every command has --help.

Configuration

Every setting goes through one registry: environment variables, a TOML file (yunshu serve --config yunshu.toml) or --set KEY=VALUE. yunshu config shows the effective values and their sources. Common ones:

All settings: Configuration.

Docs

• Clients — curl, OpenAI / Anthropic SDKs, Open WebUI, agents
• API surface and API reference
• Agent compatibility
• KV cache tiers and prompt-caching APIs
• Benchmarks and Accuracy
• Service, Troubleshooting, Changelog

Privacy

No telemetry, usage analytics or crash reports. Yunshu connects out only to download models, to web-search / MCP providers you configure, and for web fetches a request asks for.

Built on and license

MLX, mlx-lm, mlx-vlm, mlx-audio. Vendored kernels from oMLX and TensorFold keep their notices in THIRD_PARTY_NOTICES. Yunshu is Apache 2.0: LICENSE.