TurboQuant-MLX

Extreme weight and KV cache compression for LLMs on Apple Silicon. MLX implementation of Google's TurboQuant (Zandieh et al., 2025): Hadamard rotation + Lloyd-Max codebooks applied both to weights (compile time) and the KV cache (run time). Data-free: no calibration set, no retraining.

📖 Documentation: manjunathshiva.github.io/turboquant-mlx: get started, pick a model for your Mac's memory, and guides for converting, serving, coding agents, vision models and expert streaming.

Supports dense models (LLaMA, Qwen, Mistral), Mixture-of-Experts (Qwen-MoE, GPT-OSS, Qwen3.5-MoE, Qwen3.6-35B-A3B, Qwen3-235B-A22B, DeepSeek-V2, Poolside Laguna, Kimi K3 2.8T, Qwen3.8-Flash-Next 180B), Mamba/attention hybrids (Nemotron-3-Nano-4B, Nemotron-3-Super-120B, Nemotron 3.5 Super VL 120B), and multimodal models (Muse Glimmer, Qwen3.8-27B, Qwen3.8-Flash-Next, Nemotron 3.5 Super VL, and the block-diffusion DiffusionGemma). Compatible with hybrid attention architectures, attention sinks, sliding-window attention, and linear attention layers (including Gated DeltaNet).

With both weight and KV cache compression at 3-bit, GPT-OSS-120B fits its full 131K context window in 50 GB on a 64 GB MacBook — and KV cache compression actually makes generation faster on the 120B (8.7 vs 6.4 tok/s) because the smaller cache cuts memory bandwidth more than dequant costs.

Expert streaming runs MoE models whose weights exceed available RAM by paging only the router-selected experts from disk per token — e.g. the 35B-parameter Qwen3.6-35B-A3B runs on a 16 GB Mac mini in under 4 GB of RAM, with output bit-identical to the fully-resident model. See Models bigger than RAM.

Local coding — Qwen3.6-27B, a dense SWE-bench-grade coder, runs fully resident on a 48 GB Mac at 3-bit (~13 GB on disk, ~17.5 GB at runtime) and serves to Cursor / VS Code over an OpenAI-compatible endpoint. See Local coding agents.

Ternary (1.58-bit) experts shrink the biggest MoEs enough to run fully in memory: Qwen3.6-35B-A3B in 9.4 GB on a 16 GB Mac mini, Qwen3.5-122B in 30.9 GB on a 48 GB Mac, and Poolside Laguna S-2.1 118B and NVIDIA Nemotron 3.5 Super VL 120B on a 64 GB Mac. See Choose a model.

Key Results — Weight Compression

* Estimated for 0.30.0, not measured. 0.30.0 makes ternary experts decode faster (the kernels no longer do an integer division per weight). These figures come from kernel timings at each model's expert shapes, calibrated against the model's measured speeds; the same estimate predicted 20.4 tok/s for Nemotron 3.5 VL, which measured 21.0. The 0.29.0 figures in parentheses were measured.

Key Results — KV Cache Compression

KV cache compression projects to ~7 GB RAM saved at 131K context on GPT-OSS-120B and ~5 GB at 262K on Qwen3.5-122B. Roundtrip cosine similarity vs FP16: 0.983 at 3-bit, 0.995 at 4-bit.

On the Qwen3.5-122B KV rows: they used symmetric 3-bit KV on a short prompt. Fully resident with the recommended K8/V3 cache, FP16 decodes faster on that model, so there KV compression is a memory win, not a speed win; the decode speed-up is specific to GPT-OSS-120B. See KV cache compression and #19.

Community results from an Apple M5 Pro (48 GB), contributed by @sbayer2 (#14, #16), are on the Benchmarks page: resident and streaming speeds, a KV cache sweep against context, and the cache-budget sweep for the 48 GB tier.

Install

pip install turboquant-mlx-full

The package is published as turboquant-mlx-full on PyPI and imported as turboquant_mlx.

• macOS on Apple Silicon (M1 or later)
• Python 3.11+ (mlx-lm 0.32 needs it; on 3.10, pip installs turboquant-mlx-full 0.28.x)
• The Metal kernels are compiled by MLX on first use: no Xcode or CMake needed

Optional extras:

pip install "turboquant-mlx-full[vlm]"    # vision and diffusion models via mlx-vlm (>= 0.7.9)
pip install "turboquant-mlx-full[eval]"   # perplexity benchmarking (datasets, transformers)
pip install "turboquant-mlx-full[kimi]"   # Kimi K3's tiktoken-based tokenizer

From source: git clone https://github.com/manjunathshiva/turboquant-mlx.git && cd turboquant-mlx && pip install -e . (see CONTRIBUTING.md).

Quick start

1. Will it fit? turboquant-plan reads only the safetensors headers (a few hundred KB, over the network for a Hugging Face repo), so you know before a multi-GB download:

turboquant-plan --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64

Projection at 16,384 tokens of context
  weights              12.59 GB
  KV cache             0.34 GB  (20.0 KB/token, hybrid: 10/40 full-attention layers)
  prefill workspace    1.07 GB  (estimate, at --prefill-step-size 2048)
  runtime reserve      1.00 GB  (buffer cache, activations, fragmentation)
                       ----------------------------------
  peak                 15.00 GB of 61.85 GB usable   46.85 GB headroom

Verdict: ✅ RESIDENT — fits fully in memory

When a model fits only with a raised GPU memory limit or a smaller prefill step, it prints the exact sysctl line and flags. turboquant-doctor adds a read-only readiness check of the model files and machine.

2. Run it.

turboquant-generate --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
    --prompt "Why is the sky blue? Explain in simple terms." --max-tokens 300

3. Serve it over an OpenAI-compatible API at http://localhost:8080/v1, for apps, Cursor, VS Code or a coding agent:

turboquant-serve --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 --port 8080

4. Convert your own (BF16 or FP16 weights of a supported architecture; minutes, no calibration data):

turboquant-convert --hf-path meta-llama/Llama-3.2-1B --mlx-path ./llama-3.2-1b-tq3 --bits 3 --group-size 64

Documentation

manjunathshiva.github.io/turboquant-mlx

• Get started and Choose a model for your Mac's memory
• Guides: Convert a model (mixed precision, ternary experts, the streaming converter for 200B+ models) · Generate text · Serve an OpenAI-compatible API · Local coding agents · Vision models · Models bigger than RAM · KV cache compression · Memory and macOS settings
• Models, one page per family with run commands, measurements and how each build was made: GPT-OSS · Qwen3.5-122B-A10B · Qwen3.6-35B-A3B · Qwen3.6-27B · Qwen3-235B-A22B · Nemotron-3 · Nemotron 3.5 Super VL · Poolside Laguna · Kimi K3 · Sarvam MoE · Muse Glimmer · Qwen3.8-27B · Qwen3.8-Flash-Next · DiffusionGemma
• How it works · Benchmarks · Command line · Python API · Supported architectures · Changelog

How it works

1. Randomized Hadamard rotation makes any weight distribution close to Gaussian, with no data.
2. Group and scale: each group of 32, 64 or 128 weights is divided by its root-mean-square.
3. Lloyd-Max codebook: each weight snaps to the nearest centroid of the quantizer that is optimal for a Gaussian, a mathematical constant rather than something fitted.
4. Pack the indices into 32-bit words at 2, 3 or 4 bits, or as base-3 trits for the ternary {-c, 0, +c} expert tier (~1.6 bits per weight).

Fused Metal kernels decode straight from the packed form, so full-precision weights never exist in memory. The result: near-zero quality loss at 3-bit, and usable 2-bit and ternary experts on large MoEs where affine quantization breaks down. The same rotation and codebook compress the KV cache at run time. More.

Supported architectures

LLaMA, Qwen2/2.5, Qwen3.5/3.8 (text and vision), Mistral, Qwen1.5-MoE, GPT-OSS, Qwen3.5-MoE and Qwen3.6, Qwen3-MoE, Nemotron-H, DeepSeek-V2 (MLA + MoE; V3 shares the layout but is untested), and through mlx-vlm Muse Glimmer, NVIDIA Nemotron 3.5 Super VL and DiffusionGemma. Poolside Laguna, Kimi K3 and Sarvam MoE are ported to MLX here, and Qwen3.8-Flash-Next is vendored from an open mlx-lm pull request. Status and tested sizes for each: Supported architectures.

Contributing

Issues and pull requests are welcome; see CONTRIBUTING.md (commits need a DCO sign-off) and SECURITY.md for reporting vulnerabilities.

Citation

@misc{turboquant_mlx,
    title={TurboQuant-MLX: Extreme Weight and KV Cache Compression for Apple Silicon},
    year={2025},
    note={MLX implementation of TurboQuant (Zandieh et al., 2025) for both weight quantization and runtime KV cache compression}
}

License

Apache-2.0 (LICENSE, NOTICE), from v0.3.0. Versions 0.1.x–0.2.0 were released under MIT (LICENSES/MIT.txt).

Acknowledgments

• TurboQuant — Zandieh, Daliri, Hadian, Mirrokni (2025)
• MLX — Apple Machine Learning Research
• mlx-lm — MLX language model utilities