qmlx-serve 0.14.0

Disk KV restore for hybrid MoE models on Apple Silicon. A Qwen-specialised fork of qMLX that keeps a 122B model warm on a Mac.

Keeping a hybrid 122B warm on a Mac.
A Qwen-specialised fork of qMLX for long-context serving of hybrid MoE models on Apple Silicon.

Website · Docs · Blog · charted

---

Why this exists

Qwen3.5-122B-A10B is a hybrid: about 75% of its layers are DeltaNet (recurrent, linear-attention) and 25% are full attention. The recurrent state cannot be rewound to an earlier position, so a standard in-memory prefix cache drops every entry that contains those layers. On this model it missed 100% of the time (zero in-memory hits against 109 disk hits in a normal window), so we removed it. There is no in-memory prefix cache; disk is the only reuse tier.

So the only thing that keeps the model warm is disk KV restore: checkpoint the attention KV to SSD, page it back on the next turn. It is not a fallback here, it is the entire cache. qMLX is that subsystem built properly, plus the fixes needed to make it hold on real agentic-coding traffic.

The result: a follow-up question on a 130,000-token conversation goes from a multi-minute cold prefill to a sub-second restore. Measured on an M3 Ultra, a repeated 32k prompt drops from 88 seconds of prefill to 0.64 seconds, 137x faster.

What is in it

• Disk KV checkpoint and restore for hybrid recurrent + attention MoE caches, with int4 checkpoints dequantised on restore.
• Matchable-aware disk-cap eviction so the checkpoint the next turn needs never gets evicted by unmatchable interval writes.
• Honest, phase-split metrics: real decode tok/s (decode window only), real prefill throughput (excludes cached tokens), disk-restore hit rate, TTFT. No amortised (prompt+gen)/wall throughput lie.
• Live divergence logging that pinpoints the exact token where a prefix-cache match broke, so this class of bug is diagnosable in minutes.

Design principles

• Built for the Mac Studio, not portability. Optimise for Apple Silicon and unified memory. No abstraction tax to keep a CUDA path alive.
• Hybrid attention and DeltaNet are first-class. Recurrent state cannot be trimmed like a KV block, so the cache path branches on it explicitly instead of pretending it is KV-only.
• SSD cache streaming is a first-class tier, not a fallback. Unified memory is scarce. Reusable context lives on NVMe and streams back, rather than being hoarded in RAM.
• Specialise for the models you run. Qwen-first. Breadth is a cost, not a feature.
• Honest about the concurrency profile. Single-user, --max-num-seqs 1. A component that earns zero hits gets deleted, not tuned.
• Correctness beats cleverness on the cache path. A wrong restore does not throw, it corrupts. Verify the token blob byte-for-byte, quarantine bad checkpoints, prove changes on real traffic.
• Measure on the real box. Numbers come from an M3 Ultra with real models, not CI that cannot load a 122B.
• Lean by default. Minimal dependencies, no cruft.

Status

Alpha. It runs one model (Qwen3.5-122B-A10B) on one class of machine (M3 Ultra, 96GB+ unified). Qwen-first, and honest about what is built and what is not. Decode slows gradually with context because the dense-attention layers re-read a growing KV each token, but there is no cliff: it stays usable well past 100k tokens on this hardware. Windowed attention to flatten that curve further is on the roadmap.

Known limitations

• Interrupting a cold prefill discards it. A client disconnect or cancel during a long cold prefill (before the first generated token) aborts the request at 0 tokens and throws the prefill work away, so re-sending the same prompt cold-prefills again. Disk restore only helps once a prompt boundary has been checkpointed, so an interrupt-heavy workload pays a full re-prefill per interrupt. Checkpointing partial prefills at chunk boundaries so interrupted prefills retry warm is tracked in #12.

Install

uv add qmlx-serve

Or pip install qmlx-serve. The PyPI name is qmlx-serve because the exact qmlx is blocked as too similar to mlx; the import package is still vllm_mlx and the CLI is still qmlx.

From source:

git clone https://github.com/marzukia/qMLX.git
cd qMLX
pip install -e .

Serving

qmlx serve mlx-community/Qwen3.5-122B-A10B-4bit \
  --text-only --host 0.0.0.0 --port 8095 --max-num-seqs 1 \
  --enable-prefix-cache --kv-disk-checkpoint-interval 256

Drop-in OpenAI / Anthropic API, same as upstream. --text-only is required: the vision path is incompatible with the hybrid continuous-batching that the cache work depends on.

Recommended sampling

Qwen3.5-122B-A10B ships a generation_config of temperature 0.6, top_p 0.95, top_k 20, but no repetition penalty (it defaults to 1.0). With no penalty the model can loop on long generations. Add these server defaults for Qwen's recommended thinking-mode profile plus a mild repetition penalty:

  --default-temperature 0.6 \
  --default-top-p 0.95 \
  --default-top-k 20 \
  --default-repetition-penalty 1.05

These are --default-*, so a client can still override any of them per request. Keep the repetition penalty mild (1.05) so it does not degrade code output.

Credit

Forked from raullenchai/Rapid-MLX. The base engine, the OpenAI/Anthropic API surface, and the MLX serving path are theirs. qMLX adds the hybrid-aware disk restore, the eviction and metrics work, and the Qwen specialisation. We went a different direction on hybrid attention, too fundamental to reconcile in a PR, hence the fork.

Notes

The package is still imported as vllm_mlx and the CLI is still qmlx; those are kept as functional identifiers for compatibility. qmlx_* metric names, QMLX_* env vars, and the ~/.cache/qmlx/ cache path are unchanged for the same reason.