mlx-optiq 0.4.3

Mixed-precision quantization optimizer for LLMs on Apple Silicon (MLX)

mlx-optiq

Run any LLM locally on your Mac. Quantize it, serve it, and code with it.

Website: https://mlx-optiq.com  |  Docs: https://mlx-optiq.com/docs/  |  Models: https://mlx-optiq.com/models  |  Blog: https://mlx-optiq.com/blog/  |  HF org: https://huggingface.co/mlx-community

mlx-optiq is the local-LLM stack for Apple Silicon: an optimizing compiler and runtime for MLX that turns a full-precision model into the best version for a given memory and latency budget on your Mac, using per-layer sensitivity measurement instead of uniform 4-bit everywhere. The same signal drives weights, KV cache, LoRA fine-tuning, and runtime adapter swapping.

One pip install gives you three ways to work with a local model: the CLI (quantize, serve, fine-tune), OptiQ Lab (a local web workbench), and OptiQ Code (a terminal coding agent that drives your served model).

pip install mlx-optiq

What it does

• Mixed-precision weight quantization that beats uniform 4-bit at the same size. optiq convert measures each layer's sensitivity and allocates bits per layer. A static method assigns bits by architecture for models too large to measure. Methods.
• SSD expert streaming runs large MoE quants that don't fit in RAM. A 2-bit Qwen3.5-122B-A10B runs on a 36 GB Mac at ~12 GB resident, experts streamed off disk. How.
• Mixed-precision KV cache for longer context at lower memory. optiq serve runs a per-layer KV quant pipeline.
• One server, two protocols. optiq serve speaks both the OpenAI and Anthropic APIs from one process. Point Claude Code or either SDK at the same local URL.
• Speculative decoding via bundled MTP heads or paired drafters (--mtp, --drafter).
• Distributed inference across Macs. optiq cluster serve shards a model's layers across two or more Macs over Thunderbolt and exposes one OpenAI endpoint. A 2-bit Qwen3.5-122B-A10B (42.8 GiB) runs fully resident across a 36 GB + 24 GB pair at ~20 tok/s, against 4.9 tok/s streaming experts off SSD on one Mac. How.
• Sensitivity-aware LoRA (SFT + DPO) and runtime hot-swap adapters.
• OptiQ Lab (pip install "mlx-optiq[lab]" then optiq lab): a local web UI for chat, quantize, fine-tune, and dataset work. Product.
• OptiQ Code (optiq code): a terminal coding agent that drives whatever optiq serve is serving, offline, engineered for local models (never an empty patch, edit-resilient, stall-proof). Product.

Quickstart

Every mlx-optiq quant loads with stock mlx-lm:

from mlx_lm import load, generate
model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit")
print(generate(model, tok, prompt="Hello", max_tokens=50))

Installing mlx-optiq unlocks the rest. A few starting points:

# Serve with the OpenAI + Anthropic API and ~1.4x speculative decode
optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit --mtp

# Run a huge MoE that doesn't fit in RAM (experts stream off SSD)
optiq serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit --stream-experts

# Quantize a fresh model (exact sensitivity, or fast structural rules for big bases)
optiq convert Qwen/Qwen3.5-9B --target-bpw 5.0 --candidate-bits 4,8
optiq convert <large-moe> --method static --candidate-bits 2,4 --target-bpw 2.5

# Fine-tune with sensitivity-aware LoRA
optiq lora train mlx-community/Qwen3.5-9B-OptiQ-4bit --data ./jsonl_dir --rank 8

# Shard a model across two Macs over Thunderbolt (one OpenAI endpoint)
optiq cluster up                                    # on every Mac
optiq cluster serve --model mlx-community/Qwen3.5-122B-A10B-OptiQ-2bit

# Code with your local model, in your terminal (offline)
optiq code                                          # interactive, in a repo
optiq code -p "Fix the failing test in parser.py"   # headless

Full guides for serving, KV-quant, LoRA, MTP and per-family setup are in the docs. The models page lists every quant with its Capability Score, and the blog has the deeper write-ups.

Requirements

• Apple Silicon (M1 or newer), macOS, Python 3.11+.
• The published quants load with stock mlx-lm. Converting and some MoE / multimodal runtime features track mlx-lm main; install it from git when a model card asks for it.

License

MIT for the package. Quantized models follow their base model's license.