Highlights

• Lossless speculative decoding — DFlash2 or MTP drafting with batch-invariant kernels: greedy output with speculation on is token-identical to speculation off.
• Prefix cache for hybrid models — exact checkpoints for attention + GatedDeltaNet models, in RAM and on SSD, surviving restarts; optional extra storage tiers.
• The full API on the fast path — tools, JSON schema, stop, logprobs, reasoning, cancellation, across OpenAI Chat / Responses, Anthropic Messages and Ollama.
• Typed decisions — /v1/decisions answers yes/no, choice and score questions with calibrated probabilities in one forward pass of a decision checkpoint, without generating text.
• Native coding-agent support — Claude Code, Codex and opencode work through their own APIs, including server-side web search / fetch and MCP.
• Lossless by default — anything that can change output is an explicit setting.
• Local and private — no usage analytics; diagnostics never contain prompts.
• A real console — a local web console with live request phases (queued, prefill with cache-hit segments, decode), request history, logs, models, cache, a playground, API keys with per-key quotas, settings (including a CORS editor) and host power / clock / temperature, plus a floating status island. Console guide.
• Web search with citations — server-side web_search / web_fetch for Claude Code, Codex and the APIs, six providers (DuckDuckGo, Wikipedia, SearXNG, Tavily, Serper, Perplexity), a Tavily-compatible API and optional research enrichment. Guide.
• Evals and stored completions — store: true keeps chat completions locally and the OpenAI Evals API grades them with local graders. Guide.
• Host telemetry — power, GPU clock, temperatures, thermal and memory pressure, and a per-request energy estimate, unprivileged and local. Guide.
• Multiple API keys — scopes, expiry and daily quotas per key. Guide.

Capabilities

Everything Yunshu serves, grouped. Each row links to the guide with an example and the limits; items marked partial or experimental say so. The same list is machine-readable in docs/feature_index.json.

Inference

APIs

Agents & tools

Multimodal

Operations & console

Quickstart

For model selection, external storage, readiness checks and upgrades, follow the first-run guide.

Apple Silicon, macOS 14+, Python 3.13+ and uv.

uv tool install --python 3.13 "yunshu[vision]"      # or: brew install yuhuanstudio/tap/yunshu
yunshu doctor                                   # checks this Mac and says how to fix problems
yunshu pull mlx-community/Qwen3.5-9B-MLX-4bit
yunshu serve -m mlx-community/Qwen3.5-9B-MLX-4bit

The server listens on http://127.0.0.1:8000. Any OpenAI or Anthropic client works unchanged:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")  # any key works
r = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Explain MLX in one sentence."}],
)
print(r.choices[0].message.content)

from anthropic import Anthropic

client = Anthropic(base_url="http://127.0.0.1:8000", api_key="local")
msg = client.messages.create(
    model="local", max_tokens=512,
    messages=[{"role": "user", "content": "Explain MLX in one sentence."}],
)
print(msg.content[0].text)

Models live in ~/.yunshu/models (yunshu config set models_dir PATH selects the directory for future downloads; it does not move existing weights); serve -m org/name also finds the Hugging Face cache and downloads only if needed. yunshu service install -m <model> runs the server at login.

Local console

The local web console shows what the engine is doing and manages it, in English, Traditional Chinese or Simplified Chinese. It runs as its own light process (yunshu console; it never loads MLX) that yunshu serve starts next to the engine, on port 8100. It serves the console and the docs, proxies the engine API on one origin, and records the history, so it stays usable, and keeps recording, while the engine restarts or crashes. It ships inside the yunshu package (pip, uv tool and Homebrew installs alike), so there is nothing to build:

yunshu serve -m <model>
open http://127.0.0.1:8100/console/
yunshu console --engine http://mac-studio.local:8000

• Overview — live request phases, prefill and decode tok/s, prefix-cache hit segments inside the prefill bar, time to first token, Metal memory, host power, clock and temperature.
• Requests and Logs — latency distribution (cold and warm), speculative acceptance, request history, cancel; a redacted live log with search and download.
• Diagnostics — resource readouts, health checks, a Realtime probe and the support bundle.
• Models, Downloads, Cache — load, unload, warm up and fit-check models, pull from Hugging Face with progress, and inspect or clear the RAM / WARM / SSD prefix-cache tiers.
• Playground and API access — stream a chat with reasoning, compare, view the code; paste-ready setup for Claude Code, Codex, opencode, the SDKs and curl.
• Keys and Settings — API keys with quotas and usage, every engine setting (restart-required ones prompt), and a CORS editor.
• History that outlives the engine — the console process samples the engine once a second (1 s resolution for an hour, 10 s for a day, 1 min for 30 days) and keeps the request log (metadata only), so charts and request lists backfill on open, engine outages show as gaps, and nothing depends on a browser having been open. When the engine is down the console says so, keeps the docs, settings and last data, and reconnects by itself.
• A floating status island follows you across pages, and the layout works on a phone.

Full page-by-page tour with screenshots: Console guide.

Qwen3.8-27B

yunshu pull Jundot/Qwen3.8-27B-oQ4e-mtp
yunshu pull incoai/Qwen3.8-27B-DFlash2          # optional drafter, picked up automatically
yunshu serve -m Jundot/Qwen3.8-27B-oQ4e-mtp

With the drafter installed the startup log says Speculative decoding: dflash; without it the model's own MTP head drafts. yunshu doctor -m <model> reports the selected path and whether the model fits this Mac's memory.

From source

git clone https://github.com/YuhuanStudio/Yunshu.git && cd Yunshu
uv sync --extra vision
uv run yunshu serve -m <model>

To work on the console itself, run pnpm install && pnpm dev in frontend/ (see the console guide).

How it works

 OpenAI / Anthropic / Ollama clients ──► FastAPI gateway (one process)
                                           │  request validation, tool/reasoning parsing,
                                           │  server-side tools (web search / fetch / MCP)
                                           ▼
                                  engine (one MLX thread)
          ┌────────────────────────────────┴───────────────────────────────┐
   VLM batch runner (every mlx-vlm model)                 text fast path (mlx-lm models)
   shared continuous batch, per-row sampling              single-request generate_step
   speculative lane: DFlash2 / MTP / prompt-copy
   prefix cache: RAM ─► SSD ─► optional storage tiers

All GPU work runs on a single MLX thread, so requests never fight over the GPU. Each response returns as soon as its own generation ends.

Speculative decoding

For Qwen3.5-family models a single request decodes in a speculative lane:

• DFlash2 — a separate block drafter proposes several tokens per round; used automatically when a matching drafter is installed.
• MTP — the checkpoint's own multi-token-prediction head; the fallback when no drafter is present.
• Prompt-copy drafting — when the output starts repeating text from the prompt (code edits, quoting tool results, multi-turn agents), the lane proposes the continuation of that earlier text and verifies it in the same pass. It is on by default (YUNSHU_SPEC_COPY_ROWS, 0 turns it off).

Every decode and verify matmul goes through batch-invariant kernels: a token's arithmetic is the same whether it is verified alone or among other rows. That is what makes greedy output identical with speculation on and off, and sampled output exact under position-keyed sampling. Choose the drafter with YUNSHU_VLM_DRAFT (mtp, off, or a drafter path).

Prefix cache (APC)

Qwen3.5-family models mix attention layers with recurrent GatedDeltaNet layers. A recurrent state cannot be cut back to an earlier token the way a KV cache can, so ordinary prefix caching does not work. Yunshu stores exact checkpoints (KV plus recurrent state) at prefix boundaries, keyed by the text tokens and by image pixels / audio features, so a cache hit produces exactly what a cold prefill would.

Repeated or edited long prompts, multi-turn chats and agent loops restore from the nearest checkpoint instead of prefilling again. When a conversation grows, older checkpoints of the same conversation are replaced rather than piling up. yunshu cache status and yunshu cache gc show and clean the SSD caches.

Structured output

JSON schema, JSON object, regex and grammar constraints are enforced during decoding (llguidance by default), including tool-call arguments. Unsupported schema constructs are rejected with an error instead of being silently ignored.

API compatibility

Parameters covered on chat: tools / tool_choice / parallel_tool_calls, response_format (json_object, strict json_schema), stop, logprobs / top_logprobs (also streamed), n, seed, penalties, logit_bias, reasoning_effort (reasoning returned separately), streaming with usage, and cached-token counts in usage. Errors use each API's own error shape. Extensions are namespaced (x_yunshu, X-Yunshu-*), so the official SDKs ignore them. The full matrix, with how each row was verified, is in API surface.

Typed decisions

POST /v1/decisions (and the System One wire format, /v1/systemone) asks a decision checkpoint predicate, choice and ordinal-score questions about text or images. One forward pass returns typed answers with probabilities; no tokens are generated, and a non-finite result is a refusal, never a made-up probability. Supported today: Cloudflare Clef / Clef-flash MLX checkpoints. Other decision-head families (OpenJev, Laya, D1) are not loaded yet.

decision = client.decisions.create(
    model="abenzerps/Clef-MLX",
    input="The customer asks for a refund after a broken delivery.",
    questions=[{"type": "choice", "name": "route", "instructions": "Choose the support team.",
                "choices": [{"value": "support"}, {"value": "sales"}]}],
)
print(decision.answers)  # typed value, confidence, every option's probability

Also new on main for 0.1.5: stored chat completions, Evals (/v1/evals) and Realtime client secrets.

Decisions, Evals, web search and Tavily API

Coding agents

yunshu launch claude      # or: codex, opencode

yunshu launch writes the client configuration (base URL, model, context window and output limits, reasoning effort) and starts the agent. For Claude Code it also installs a status line showing live prefill progress, decode speed and cache hits.

• Claude Code — Messages API with streaming, thinking, count_tokens for /context, model discovery, and its WebSearch tool, which Yunshu runs server-side.
• Codex — Responses API with reasoning items, function calls, local compaction and web_search.
• opencode — Chat Completions with tools and usage.

Server-side web search defaults to best-effort DuckDuckGo HTML and Wikipedia; queries leave the machine. YUNSHU_WEB_SEARCH_PROVIDER=none disables it. Auto metasearch runs configured keyed providers in parallel with the keyless sources; SearXNG is optional. MCP servers named in a request are connected by the gateway. What each agent calls and how it was checked is in Agent compatibility.

Performance

Qwen3.8-27B (oQ4e) on an M5 Max, 128 GB, single request, greedy. Methods, raw results and the full comparison tables are in docs/BENCHMARKS.md.

Snapshot of main on 2026-10-02/03 (unreleased after v0.1.4; the table's TensorFold column is the 2026-10-02 same-window run); a fresh same-window 0.1.4 comparison is pending. TensorFold is still faster at single-request decode and at 32K follow-up turns; closing those gaps is the main ongoing work. Structured output keeps speculative decoding (23 tok/s without it). Speculation never changes Yunshu's greedy output. Accuracy against the stock MLX path is checked at three levels (logit alignment, greedy divergence, paired downstream evals) in Accuracy.

Models

/v1/models returns each model's card: context length, output limit, and which inputs and features it actually supports.

Other capabilities

Command line

Every command has --help.

Configuration

Every setting goes through one registry: environment variables, a TOML file (yunshu serve --config yunshu.toml) or --set KEY=VALUE. yunshu config shows the effective values and their sources. Common ones:

All settings: Configuration.

Docs

• Console — every page, with screenshots; CLI — first run, models, launchd, JSON and shell completion
• Inference features, Multimodal endpoints, Model support
• Clients — curl, OpenAI / Anthropic SDKs, Open WebUI, agents
• API surface, API extensions, API reference, Transports
• Decisions, Evals, Web search, Tavily API
• Agent compatibility
• Authentication, keys, settings and CORS, Telemetry
• KV cache tiers and prompt-caching APIs
• Benchmarks and Accuracy
• First run, Service, Troubleshooting, Configuration, Changelog

Privacy

No usage analytics or crash reports are sent. Host telemetry (power, thermal, memory) is sampled locally for the console and never leaves the machine (guide). Yunshu connects out only to download models, to web-search / MCP providers you configure, and for web fetches a request asks for.

Built on and license

MLX, mlx-lm, mlx-vlm, mlx-audio. Vendored kernels from oMLX and TensorFold keep their notices in THIRD_PARTY_NOTICES. Yunshu is Apache 2.0: LICENSE.