mlx-optiq 0.3.4

Mixed-precision quantization optimizer for LLMs on Apple Silicon (MLX)

mlx-optiq

Quantize, fine-tune and serve LLMs entirely on Apple Silicon.

Website: https://mlx-optiq.com  |  Docs: https://mlx-optiq.com/docs/  |  Models: https://mlx-optiq.com/models  |  Blog: https://mlx-optiq.com/blog/  |  HF org: https://huggingface.co/mlx-community

mlx-optiq is an optimizing compiler and runtime for MLX. It turns a full-precision model into the best version for a given memory and latency budget on your Mac, using per-layer sensitivity measurement instead of uniform 4-bit everywhere. The same signal drives weights, KV cache, LoRA fine-tuning, and runtime adapter swapping.

pip install mlx-optiq

What it does

• Mixed-precision weight quantization that beats uniform 4-bit at the same size. optiq convert measures each layer's sensitivity and allocates bits per layer. A static method assigns bits by architecture for models too large to measure. Methods.
• SSD expert streaming runs large MoE quants that don't fit in RAM. A 2-bit Qwen3.5-122B-A10B runs on a 36 GB Mac at ~12 GB resident, experts streamed off disk. How.
• Mixed-precision KV cache for longer context at lower memory. optiq serve runs a per-layer KV quant pipeline.
• One server, two protocols. optiq serve speaks both the OpenAI and Anthropic APIs from one process. Point Claude Code or either SDK at the same local URL.
• Speculative decoding via bundled MTP heads or paired drafters (--mtp, --drafter).
• Distributed inference across Macs. optiq cluster serve shards a model's layers across two or more Macs over Thunderbolt and exposes one OpenAI endpoint. A 2-bit Qwen3.5-122B-A10B (42.8 GiB) runs fully resident across a 36 GB + 24 GB pair at ~20 tok/s, against 4.9 tok/s streaming experts off SSD on one Mac. How.
• Sensitivity-aware LoRA (SFT + DPO) and runtime hot-swap adapters.
• OptiQ Lab (pip install "mlx-optiq[lab]" then optiq lab): a local web UI for chat, quantize, fine-tune, and dataset work.

Quickstart

Every mlx-optiq quant loads with stock mlx-lm:

from mlx_lm import load, generate
model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit")
print(generate(model, tok, prompt="Hello", max_tokens=50))

Installing mlx-optiq unlocks the rest. A few starting points:

# Serve with the OpenAI + Anthropic API and ~1.4x speculative decode
optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --mtp

# Run a huge MoE that doesn't fit in RAM (experts stream off SSD)
optiq serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit --stream-experts

# Quantize a fresh model (exact sensitivity, or fast structural rules for big bases)
optiq convert Qwen/Qwen3.5-9B --target-bpw 5.0 --candidate-bits 4,8
optiq convert <large-moe> --method static --candidate-bits 2,4 --target-bpw 2.5

# Fine-tune with sensitivity-aware LoRA
optiq lora train mlx-community/Qwen3.5-9B-OptiQ-4bit --data ./jsonl_dir --rank 8

# Shard a model across two Macs over Thunderbolt (one OpenAI endpoint)
optiq cluster up                                    # on every Mac
optiq cluster serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit

Full guides for serving, KV-quant, LoRA, MTP and per-family setup are in the docs. The models page lists every quant with its Capability Score, and the blog has the deeper write-ups.

Requirements

• Apple Silicon (M1 or newer), macOS, Python 3.11+.
• The published quants load with stock mlx-lm. Converting and some MoE / multimodal runtime features track mlx-lm main; install it from git when a model card asks for it.

License

MIT for the package. Quantized models follow their base model's license.