VeloxQuant-MLX 0.59.0

43 compression methods — quantizers, token-eviction caches, cross-layer merging — in MLX

Docs · Playground · Contributing

VeloxQuant-MLX shrinks the KV cache of any mlx_lm model on Apple Silicon, up to 16× smaller with near-lossless quality, in three lines of code. If you run models locally and keep hitting a context-length or memory wall, you swap in a compressed cache and change nothing else about the model.

Inside are 43 compression methods, each adapted from a published paper, spanning zero-calibration 1-bit quantizers, token-eviction caches, and cross-layer merging. The hottest path also has hand-written Metal kernels behind it, up to 14.7× faster than pure MLX. The project is actively developed and every release is gated on the full test suite (see badges above).

Accounting vs. resident memory: the compression ratios above are the theoretical byte count (bit-width accounting), not what Activity Monitor will show you. Most quantization methods still store full fp16 tensors internally on the default mlx_lm serving path today, so process RSS won't drop by the same factor yet. See #27 for the packed-storage roadmap. Methods that do shrink resident memory today (eviction/merging, which actually drop tokens) are marked 🔻RSS in the method library below; the rest reduce accounting-only storage while staying fp16-sized in memory.

All 43 strategies share one 3-line API, so switching between them just means changing method="..." rather than rewriting your code. The hot path runs on hand-written Metal kernels (6.9-14.7× faster quantize, 98% less peak memory at the shape that used to OOM), and where a method had to cut a corner to work as a drop-in cache instead of a full model rewrite, its docs page says so rather than papering over it. It's validated on 12 production models (Llama, Mistral, Qwen, Phi, Gemma 3/4, Falcon), and vision-language models work too: patch_vlm_kv_cache wires the same caches into mlx-vlm single-prompt generation (Qwen2-VL, LLaVA, and others). Docs.

import mlx_lm
from veloxquant_mlx import KVCacheBuilder, KVCacheConfig

model, tokenizer = mlx_lm.load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")
config = KVCacheConfig(method="turboquant_rvq", bit_width_inlier=1, seed=42)
caches = KVCacheBuilder.for_model(model, config)
model.make_cache = lambda *_a, **_k: caches

response = mlx_lm.generate(model, tokenizer, prompt="Explain relativity simply.", max_tokens=200)

Numbers

Compression ratios below are bit-width accounting, not measured RSS (see the accounting-vs-resident note above and #27). "Peak memory reduction" and "context at 8 GB" rows are Metal-kernel working-set/estimate figures, not steady-state cache RSS under default mlx_lm serving.

Table of contents

1. Installation
2. Quickstart
3. Method library — all 43 methods, grouped by family
4. Metal kernels
5. Benchmark results
6. What's inside
7. Architecture
8. CLI
9. Development
10. Project & governance — security policy, maintainership, release process
11. Documentation & blog posts
12. References
13. Support

Installation

pip install VeloxQuant-MLX

Requirements: Apple Silicon M1+, Python ≥ 3.11, MLX ≥ 0.18, NumPy ≥ 1.26.

Source install, conda/miniforge, Metal troubleshooting, and verifying the install are covered in the installation guide.

Quickstart

No Python? Start here — the control panel

veloxquant panel

That opens a local web UI at http://127.0.0.1:7860. Pick a model and a compression method, press Start Server, and point any OpenAI-compatible client (Claude Code, Cursor, the OpenAI SDK) at the URL it gives you.

The panel drives veloxquant serve, which you can also use directly:

veloxquant methods --servable-only        # what can be served
veloxquant serve --model mlx-community/Llama-3.2-1B-Instruct-4bit \
                 --method turboquant_rvq --bits 2 --port 8000

Compression is currently accounting-only: byte counters measure compression fidelity, not runtime memory saved. See #27.

Full guide: docs/control-panel.md

RVQ 1-bit — 7.5× compression, no calibration (recommended default)

import mlx_lm
from veloxquant_mlx import KVCacheBuilder, KVCacheConfig

model, tokenizer = mlx_lm.load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")

config = KVCacheConfig(method="turboquant_rvq", bit_width_inlier=1, seed=42)
caches = KVCacheBuilder.for_model(model, config)
model.make_cache = lambda *_a, **_k: caches

response = mlx_lm.generate(
    model,
    tokenizer,
    prompt="Explain the theory of relativity in simple terms.",
    max_tokens=200,
)

The 5-minute quickstart walks through the example above, then moves to VecInfer (16×, Metal-accelerated) as a stronger-algorithm follow-on. Two other guides worth knowing about:

• Mixed-precision guide, covering RateQuant's automatic per-layer bit allocation via reverse-waterfilling
• mlx_lm integration guide, for wiring compressed caches into any model

Method library

Every one of the 43 methods drops in the same way: set method="<id>" in KVCacheConfig. The algorithm overview has the full comparison table, a decision tree, per-model recommendations, and for each method its mechanism, config, evidence, and limitations.

If you want a starting point:

• No calibration, best default → turboquant_rvq b=1 (7.5×, 0.92 cosine)
• Max compression, Qwen2.5/Gemma → vecinfer 1-bit (16×, Metal-accelerated)
• Best quality at moderate compression → spectral b=3 (5.33×, ~5s calibration)
• Heterogeneous layers (sensitivity ratio >2×) → RateQuant on top of RVQ
• Max context length, fixed RAM → rabitq keys + MSE-b4 values (6× full KV)
• RoPE-compatible exact VQ → comm_vq (ICML 2025, 64× key compression)
• Hard cap on token count, fixed RAM → h2o or snapkv (eviction, reduces resident memory)

The 43 methods fall into three families, each entry linking to its docs page:

• Quantization (22 methods) — compress every token. Default: TurboQuant RVQ turboquant_rvq. Also: VecInfer, SpectralQuant, RateQuant, RaBitQ, QJL, PolarQuant, CommVQ, KIVI / KIVI-Sink, SKVQ-adapted, SVDq, Kitty, KVQuant-NUQ, NSNQuant-adapted, ZipCache-adapted, GEAR, CacheGen, AMC-adapted, A2ATS-adapted, AnchorKV-adapted.
• Low-rank & cross-layer (6 methods) — compress across dimensions or depth. PALU, XQuant, MiniCache, xKV-adapted, AdaKV-proxy, KVTC-adapted.
• Token eviction & merging (15 methods) — drop or merge low-value tokens; these reduce resident memory today (🔻RSS), not just accounting. SnapKV-adapted, StreamingLLM-adapted, H2O-adapted, TOVA-adapted, PyramidKV-adapted, SqueezeAttention-adapted, ChunkKV-adapted, CaM-adapted, L2Norm-adapted, Q-Filters-adapted, Keyformer-adapted, MorphKV-adapted, KVzip-adapted, CurDKV-adapted, NestedKV-adapted, RocketKV-adapted.

🔻RSS in the family lists above means the method actually reduces memory you can see today (it drops tokens or stores a genuinely smaller tensor); everything else there only shrinks the theoretical bit count while still storing full fp16 internally (see #27). Also worth knowing: every "-adapted" method is an adaptation, not a 1:1 port. The cache only sees per-layer K/V, never the model's real query/attention maps, so attention-based signals fall back to a key-as-query proxy. Per-method compression ratios, categories, and release versions are on the algorithm overview.

Metal kernels

VecInfer's quantize_vq was the slowest step in the pipeline, so it now runs on the GPU instead of in Python. It's a 30-line Metal shader, JIT-compiled by mx.fast.metal_kernel the first time you call it, with the same Python API. No code changes needed to benefit.

_{Benchmarked on Apple Silicon GPU. Left: quantize latency. Center: speedup factor. Right: peak memory.}

The memory win comes from what never gets written out. The pure-MLX version has to materialize a [N, n_centroids, sub_dim] diff tensor, but the kernel skips it entirely because the argmin accumulator lives in thread-local GPU registers. That's the whole 98% peak-memory drop.

Caveat: the kernel pays a ~50–200 µs launch overhead per call. On tiny models (SmolLM2-135M, ~60 launches/token) that overhead can exceed the savings. Built for the regime that needs it: 7B+ models at realistic context lengths.

Full kernel source and how it was built: blogs/metal-kernels.md. Usage, fallback behaviour, and debugging: docs — Metal GPU kernels.

Fused RaBitQ asymmetric pipeline

Two newer kernels form a fully GPU-resident pipeline for an asymmetric-precision cache: 1-bit packed keys scored via XOR+popcount, 4-bit codebook values. That K/V format combination is one fused attention kernels normally can't express.

• rabitq_encode does rotate + binarize + bit-pack + magnitude in one dispatch. Sign packing uses simd_ballot: each SIMD-group's 32 sign predicates land in a single vote mask, which is exactly 4 bytes of packed output.
• rabitq_fused_attend scores packed keys, runs an online softmax split across 8 SIMD-groups (flash-decoding style), and accumulates codebook values, all in one dispatch. No dequantized K or V is ever materialized.
• rabitq_pack_values packs two 4-bit value indices per byte. The attend kernel reads nibbles directly (auto-detected from the shape), which halves value-cache memory and bandwidth with bit-identical outputs.

Measured on Apple M4, D=128 (scripts/metal_rabitq_attend_bench.py, scripts/metal_rabitq_encode_bench.py):

Caveat: with unpacked (byte-per-index) values the fused attend loses at short contexts (0.65× at S_kv=512). Nibble-packing halves value bandwidth and flips that to a small win.

Parity vs numpy references is covered by 63 dedicated tests (test_rabitq_attend.py, test_rabitq_encode.py, test_rabitq_values.py), including an end-to-end encode→attend test and bit-exact packed-vs-unpacked equality.

For large S_q, meaning the multi-turn VLM case where a new turn attends over a long compressed image-token history, rabitq_prefill_attend is the matmul-shaped companion: both Q·K̂ᵀ and W·V̂ run on 8×8 simdgroup_matrix tiles, with keys sign-decoded and values nibble-decoded inside the tile loop. It scores exact dots rather than the Hamming estimate, and is cross-attention only (no causal mask).

Fused group-affine (KIVI-style) attention — new in 0.42.0

scalar_fused_decode_attend is the scalar/group-quant analogue of the codebook fused attends above. It serves the KIVI / SKVQ / Kitty / group-quant family, where K/V are uint8 codes plus a per-group (scale, zero) pair instead of a codebook.

The pure-MLX path pays a real cost every decode step: it reconstructs code * scale + zero into a full fp16 tensor, writes that to memory, then reads it back for scaled_dot_product_attention (a dequantize → DRAM → SDPA round-trip). This kernel skips that round-trip entirely, reconstructing x_hat directly in GPU registers inside a FlashAttention-style online softmax (a numerically stable way to compute softmax over a stream of values without holding them all in memory at once). No dequantized K_hat/V_hat ever touches DRAM.

The win grows with context length: the fp16 K_hat the old path builds grows linearly with S_kv, while the packed codes this kernel reads directly stay 16/b times smaller.

Measured (Apple M4 10-core GPU, B=1 H=32 D=128 b=2 g=32 S_q=1) vs. dequantize → MLX SDPA:

The kv axis is split flash-decoding style across nsg SIMD-groups so single-query decode shapes still fill the GPU (nsg=8 tuned on M4), and one compiled kernel serves any (S_kv, D, g). Parity max abs error is 1.2e-4; the fp32 softmax accumulation makes it more accurate than the fp16 baseline it replaces (test_scalar_attend.py).

Benchmark results

10-model comparative study — VecInfer vs RVQ (v0.5.0)

_{End-to-end mlx_lm.generate · 200-token prompt · 120-token generation · Apple M-series unified memory}

Compression ratio:

Throughput (tok/s):

RVQ-1bit is the safe default: within 5% of fp16 on most 7–8B models, with zero calibration. VecInfer-1bit wins on memory (always 16×) and on throughput for strong-GQA models (Qwen2.5, Gemma).

Historical benchmark snapshots (throughput optimisation journey, RateQuant V2, 8-model RVQ sweep) and full methodology: BENCHMARK_RESULTS.md.

What's inside

Full module reference and API docs: docs — API reference.

Architecture

Every method runs the same three-step pipeline: rotate the K/V tensors into a friendlier basis, quantize them (optionally with a residual pass for extra precision), then pack the bits. That is why swapping method="..." just works. Every quantizer plugs into the same KVCacheConfig → KVCacheBuilder → mlx_lm-compatible cache path regardless of what it does internally.

The wiring underneath is conventional object-oriented plumbing, plus some custom data structures for the bit-packing. Pipeline diagrams (TurboQuantRVQ, VecInfer) and the design-pattern breakdown are in docs — Core concepts.

CLI

# Precompute rotation matrices, JL matrices, codebooks
python -m veloxquant_mlx precompute \
    --head_dim 128 --bits 1 2 3 4 --jl_dim 128 --seed 42 \
    --output_dir ./artifacts/

# Synthetic benchmark — single config
python -m veloxquant_mlx benchmark \
    --method turboquant_rvq --head_dim 128 --bits 2 --seq_len 1000

# End-to-end model benchmarks
python benchmark_scripts/benchmark_vecinfer.py   # VecInfer 10-model sweep
python benchmark_scripts/run_outlier_ratequant.py # RateQuant mixed-precision

# Which method should I use on my Mac? (new in 0.42.0)
python -m veloxquant_mlx recommend \
    --chip M4 --ram-gb 16 --model-class 7B --goal everyday

The recommender is accounting-aware. It reports the key compression ratio and also tells you when resident RAM savings are unlikely, rather than quoting a ratio that won't show up in RSS:

method=turboquant_rvq
knobs={'bit_width_inlier': 1, 'seed': 42}
key_accounting_ratio≈7.5x
resident_savings_likely=False
kv_fp16_mb≈512.0  kv_compressed_mb_est≈68.27
rationale: The safe everyday pick: it works out of the box with no setup
           step, and shrinks the key half of the cache by about 7.5x. It
           unpacks each value back to full precision as it is read, so this
           is a size measurement rather than a drop in live memory use.
warnings:
  - RAM is tight for a model this size. For long prompts you will get more
    out of 'Fit the longest conversation' (rabitq), which compresses the
    whole cache, or 'Never grow past a fixed memory limit' (streaming_llm),
    which caps it outright.
  - This method measures smaller but may not free much actual RAM on short
    prompts, because its default path unpacks values back to full precision
    as it reads them. The size figure is real; treat it as a measure of how
    well the data compresses, not as RAM you get back.

Goals: everyday, max_key_accounting, max_context, best_quality, constant_memory. Add --json for machine-readable output, or --seq-len / --n-layers / --n-kv-heads / --head-dim to match a specific model. Also available in the browser via the Compression Lab.

Load precomputed artifacts to skip re-computation at runtime:

from veloxquant_mlx.artifacts import NpyArtifactStore

cache = (
    KVCacheBuilder()
    .with_method("turboquant_rvq")
    .with_head_dim(128)
    .with_bit_width(inlier=2)
    .with_artifact_store(NpyArtifactStore("./artifacts/"))
    .build()
)

Development

# Full test suite (includes Metal parity tests)
pytest veloxquant_mlx/tests/ -v

# 2-bit improvement validation — fast synthetic run
python benchmark_scripts/test_2bit_improvements.py

# Generate optimization-journey figure
python scripts/plot_optimization_journey.py

Contributions welcome. Open an issue first for anything beyond a small bugfix. See CONTRIBUTING.md for guidelines and CHANGELOG.md for release history.

Project & governance

These policies already governed the project. This section makes them reachable from the README rather than only from the file tree.

I maintain this project (rajveer43), with one other contributor active on the codebase. That is a key-person risk: if you are evaluating VeloxQuant-MLX for production or funding, weigh it accordingly. GOVERNANCE.md describes how co-maintainers are added, and the MIT license means the code stays usable and forkable either way. Expanding the maintainer group and writing down release-critical knowledge is open work, tracked in issues.

Lint and the non-Metal unit suite run on GitHub Actions (Ubuntu, Python 3.12) for every push. The full suite, including Metal parity tests that need Apple Silicon, gates each release. Metal kernel correctness is covered by dedicated numpy-parity tests, not benchmarks alone; see Metal kernels.

Documentation & blog posts

Full docs, including per-method pages, guides, and API reference: https://veloxquant-mlx.netlify.app/

Deep-dive writeups live in blogs/ and are also published on the docs site:

Beyond compression: cross-model KV transfer

One capability in this repo is not a compression method and is deliberately not counted in the 43: cross-model KV cache transfer (veloxquant_mlx.transfer). Instead of shrinking one model's cache, it maps a source model's already-prefilled KV into a target model's format, so the receiver can skip prefill when you swap between two models in the same family. Cache size is unchanged; what you save is prefill compute.

It lives in its own subsystem rather than behind method="..." because it needs two models, an offline per-pair fit, and a multi-GB artifact, none of which the single-model cache contract can express. Adapted from Cross-Model KV Cache Transfer (NVIDIA, arXiv:2608.03893). The paper's retention and speedup figures are its own, measured on datacenter-scale pairs, and are not reproduced here. Read the docs page for the caveats before relying on it.

References

43 methods, each adapted from a published paper with documented deviations. 39 come from a verified peer-reviewed venue; 2, NestedKV-adapted and AMC-adapted, come from unpublished preprints as one-time, stated exceptions (see CITATIONS.md). The full bibliography, covering implemented methods, related work, and survey papers, is in CITATIONS.md. The cross-model transfer subsystem above is counted separately since it compresses nothing.

Headline references: TurboQuant (ICLR 2026), VecInfer (2024), RaBitQ (SIGMOD 2024), CommVQ (ICML 2025), KVzip (NeurIPS 2025), KVTC (ICLR 2026), CurDKV (NeurIPS 2025), NestedKV (preprint, arXiv:2605.26678), AMC (preprint, arXiv:2607.10109), A2ATS (ACL 2025 Findings). Built on Apple MLX.

Support

VeloxQuant-MLX is free and MIT-licensed. There is no commercial offering and no revenue behind it. Funding is not required to use it and never gates a feature; it buys maintenance time.

• Stars, issue reports, and PRs are the most useful contribution. They tell me what breaks on hardware I do not have.
• Want to help fund it? Buy me a chai ☕ or tip on Ko-fi 💜.
• Found a vulnerability? Do not open a public issue; follow SECURITY.md instead.
• Citing this work: DOI 10.5281/zenodo.20647294, with per-method attributions in CITATIONS.md.

License

MIT — see LICENSE.