pion-vllm-mlx

The Pion prompt cache for mlx-lm on Apple Silicon: a prefix's K/V that a different process, a different model object, or a restarted server can reuse (Stage 1), plus an attention patch that lets Pion compute the attention over that prefix itself (Stage 2).

Time to first token on Llama-3.2-1B-4bit with a 2,049-token prefix and a 16-token question: 1,193 ms cold in vanilla mlx-lm, 46.2 ms warm in the same process (26×), and 69.0 ms from a separate process over the wire (17×). A shorter prefix saves less: 12× at 1,035 tokens, 4.6× at 268, and 1.4× at 34, where the saving is about 14 ms. Harness: cross_process_ttft.py; raw output of every run: cross_process_ttft_2026_10_07.json.

mlx-lm's own prompt-cache file is faster. save_prompt_cache / load_prompt_cache answer the same question in 37.0 ms from a fresh process, mapping a 67 MB file (file_cache_ttft.py, raw output). If one program reuses one fixed prefix, use the file. This package is for what a file does not do: a cache any process or model object reads over the Redis wire, an acked write that survives a crash, and, in pion-vllm-mlx serve, a longest-prefix match over every stored conversation that a restarted agent or a second session picks up without naming a file.

From another process on the same machine, a hit maps the prefix as a file the server wrote when it was stored (V.EXPORT), the way mlx-lm's load_prompt_cache reads its own file, instead of copying it over loopback TCP; the measured row is in Pion's README. A remote server, a cache that is not a plain KVCache, or PION_PROMPT_CACHE_NO_EXPORT=1 keeps the TCP lane.

Stage 2 runs on three lanes, auto-selected by PionPromptCache:

Measured with tests/bench_w1_stage2.py on an M4 Mac mini, 2026-10-07; raw output in benchmarks/results/2026-10-07-mac-m4/.

Plus HybridRetrievalCache for RAG: chunk-id-keyed K/V hydration (3.5× p50 TTFT and 96.1% token agreement on a 100-query SQuAD v2 run, stage1_hybrid_recall_bench.py, raw results).

Install

pip install 'pion-vllm-mlx[mlx]'

It talks to a Pion server started with the prompt cache enabled:

brew install pavelhorak/tap/pion && brew services start pion   # macOS; the service runs with these flags
./pion-server --kvcache --metal-attention                       # or from a release tarball / source build

pion-vllm-mlx serve: one local endpoint for coding agents

pion-vllm-mlx serve --model mlx-community/Qwen3-4B-4bit --port 8080
export ANTHROPIC_BASE_URL=http://127.0.0.1:8080      # Claude Code
export OPENAI_BASE_URL=http://127.0.0.1:8080/v1      # Aider, Continue, Zed; Codex: wire_api = "responses"

mlx-lm's server with its prompt cache in Pion, plus Anthropic /v1/messages (Claude Code) and OpenAI /v1/responses (Codex). A restarted serve, a second session and a second serve process start from the stored prefix instead of a full prefill, with the same greedy reply. Within one live session it does what stock mlx-lm does, and hybrid models (sliding-window or SSM layers) are not stored. Measured on one recorded Claude Code session against stock mlx-lm, Ollama, LM Studio and oMLX, only oMLX and serve skip the re-prefill after a restart; oMLX 0.7 also does it on hybrid models, which serve does not. The console script arrived after 0.1.5; on 0.1.5 run python -m pion_vllm_mlx.serve with the same flags. Guide: doc/coding_agents.md.

The mlx extra pins mlx-lm>=0.20.1,<0.33. That ceiling is not decoration: install_pion_attention_patch() replaces a private mlx-lm function and rebinds the snapshot-bound name inside every imported mlx_lm.models.* module. The signature is re-checked at patch time, so an incompatible mlx-lm raises PionMlxCompatError naming the observed signature and patches nothing, rather than failing as a TypeError inside your generation loop. To see the seam:

python -c "import pion_vllm_mlx as p; print(p.mlx_lm_seam_report())"

From a checkout of the Pion repository, pip install -e 'pion-vllm-mlx/[dev,mlx]' adds the dev tools, then python pion-vllm-mlx/tests/test_mlx_lm_seam.py checks one mlx-lm version and pion-vllm-mlx/tests/run_mlx_version_matrix.sh the whole matrix.

The MLX dependency is optional because the package can act as a wire-compatibility shim on machines that don't have MLX (e.g. a Linux test runner). Lanes 2 and 3 work without MLX in the consumer; lane 1 requires it.

Lanes 2 and 3 need the server started with --kvcache --metal-attention, as above.

Usage

The four lines (Stage 1 — cross-process, native decode)

from mlx_lm import load, generate
from pion_vllm_mlx import PionPromptCache

model, tok = load("mlx-community/Llama-3.2-1B-Instruct-4bit")
prefix_ids = tok.encode(SYSTEM_PROMPT, add_special_tokens=False)   # the part every request shares
suffix_ids = tok.encode(USER_TURN, add_special_tokens=False)       # the part that changes

pc = PionPromptCache(model, host="127.0.0.1", port=1974)                       # once per loaded model
cache = pc.get_or_prefill(prefix_ids, namespace="app|v1|llama-1b|system_v1")  # MISS: prefill once + store · HIT: fetch
text = generate(model, tok, prompt=suffix_ids, prompt_cache=cache)             # decode as usual, at native speed

get_or_prefill returns what mlx_lm.make_prompt_cache(model) would, with the prefix's K/V already in it, so mlx-lm decodes at native speed. The namespace names one exact token sequence for one model and quantization — key it on the tokens, never the text, and change it when either changes. The first process to ask pays the prefill once; every later call, from any process, fetches it. Runnable with timings: examples/prompt_cache_demo.py.

Stage 2 — Pion computes the attention over the prefix

from mlx_lm import load, generate
from pion_vllm_mlx import PionPromptCache, install_pion_attention_patch, make_pion_prompt_cache

install_pion_attention_patch()                                          # once per process
model, tok = load("mlx-community/gemma-4-e2b-it-4bit")
pc = PionPromptCache(model, stage2=True, host="127.0.0.1", port=1974)
ns = "rag/corpus_v1"

if not pc.lookup(ns):                       # cold: prefill once, push per-layer K/V to Pion
    pc.get_or_prefill(prefix_ids, namespace=ns)

# warm: no prefill anywhere; attention over the prefix runs in Pion
cache = make_pion_prompt_cache(model, namespace=ns, prompt_cache=pc, prefix_len=len(prefix_ids))
text = generate(model, tok, prompt=suffix_ids, prompt_cache=cache)

When the cold prefill ran in this same process the prefix stays resident as MLX arrays (lane 1, zero wire round trips — the 46.2 ms / 26× same-process row in the main README). From any other process the patch uses lane 2 or 3, and then every decode step pays one round trip per layer. So for a consumer that only generates text from another process, the four Stage-1 lines above are the faster end-to-end path; Stage 2 is for consumers that want the attention itself in Pion — the sparse long-context selectors (sparse_full_layers=), pion-exo, a custom vLLM CacheEngine. stage2=True without make_pion_prompt_cache changes nothing except a slightly slower cold path.

Hybrid models supported (per-layer routing): Llama-3.2-1B-Instruct-4bit, Qwen3.5-4B, Gemma-4-E2B-it-4bit, Gemma-4-12B-it-4bit. Add a new hybrid arch by exposing model.args.layer_types + sliding_window — the patch reads both directly.

Docs

• Main project: github.com/pavelhorak/pion · website: pion.pavelhorak.com
• Shared KV cache design: doc/shared_kv_cache.md
• 64K NIAH reproducer (Gemma 4): examples/sparse_mask_64k_niah.py

License

Apache-2.0 — see LICENSE. Pion's client packages are deliberately permissive so they can be vendored into any stack; the Pion server this package talks to is Apache-2.0 too, with one closed binary library for its tuned vector kernels. The full map of what is under which licence is doc/licensing.md.