gguf2mlx — convert GGUF to MLX without guessing

Why this exists

GGUF is a distribution format; MLX-LM and mlx-vlm expect a Hugging Face-style directory with config.json, tokenizer assets and safetensors weights. gguf2mlx bridges that gap for explicitly supported architectures by reading GGUF metadata and tensors, rebuilding the MLX-compatible layout, and failing closed when a model layout is not actually supported — no silent "Llama fallback".

Use it when a model ships GGUF but no MLX checkpoint, or when you need an inspectable HF-style directory on an M1–M4 Mac. If the original Hugging Face weights are available, prefer mlx_lm.convert — it avoids inheriting the source GGUF's quantization error.

Conversion itself only needs numpy, gguf and safetensors, so it runs fine on Linux servers and CI; MLX is required only for --quantize.

Important: the default output is FP16/BF16 safetensors. --quantize re-quantizes from that FP16 copy. To keep the bytes that are already in the GGUF, use --quant-source preserve, which re-expresses the source blocks as MLX affine weights for llama, qwen2, qwen3 and mistral (see Preserve GGUF quantization).

Install

pip install gguf2mlx            # installs MLX + MLX-LM on Apple Silicon
pipx install gguf2mlx           # isolated CLI
uv tool install gguf2mlx        # or with uv

Extras: [vlm]/[publish] add Hugging Face Hub, [all] everything, [dev] the test/lint toolchain.

Quick start

# GGUF -> FP16 MLX-LM directory
gguf2mlx --input model.gguf --output ./mlx-model

# GGUF -> 4-bit MLX-LM output
gguf2mlx --input model-Q4_K_M.gguf --output ./mlx-model-4bit \
  --quantize --q-bits 4 --q-group-size 64

# Bounded-memory direct quantization (no FP16 staging directory)
gguf2mlx --input model-Q4_K_M.gguf --output ./mlx-model-4bit \
  --quantize --direct-quant --q-bits 4 --q-group-size 64 --q-mode affine

# Reuse the GGUF blocks instead of quantizing the FP16 copy again
gguf2mlx convert --input model-Q4_K_M.gguf --output ./mlx-model-preserved \
  --quant-source preserve

# Inspect metadata without writing weights
gguf2mlx --input model.gguf --skip-weights

# Smaller shards (payload bytes, not source bytes)
gguf2mlx --input model.gguf --output ./mlx-model --max-shard-gb 2

# Package / verify a directory as a .mlx bundle
gguf2mlx package --model-dir ./mlx-model --output ./mlx-model.mlx
gguf2mlx verify --bundle ./mlx-model.mlx

Load the result:

from mlx_lm import load, generate
model, tok = load("./mlx-model")
print(generate(model, tok, prompt="Hello from MLX", max_tokens=32))

See How conversion works for streaming, sharding and offline tokenizer reconstruction.

Is the output trustworthy?

Every conversion writes conversion_report.json and gates on weight fidelity: each written tensor is compared position-wise against an independent decode of the source GGUF, with a per-dtype relative-L2 tolerance. Fused or permuted tensors are reported as transformed rather than silently counted as verified.

# Re-audit an existing artifact against the GGUF it came from
gguf2mlx verify --model-dir ./mlx-model --weights --source model.gguf

# Load it with mlx_lm and require finite logits from one forward pass
gguf2mlx verify --model-dir ./mlx-model --load

# Skip the (roughly 2x) audit cost
gguf2mlx convert --input model.gguf --output ./mlx-model --no-weight-audit

Perplexity is recorded but never gated: cross-runtime PPL mixes quantization noise with conversion error, so it cannot prove a converter correct. Nightly, the pinned load matrix converts real GGUF models and proves each one loads and runs.

Preserve GGUF quantization

--quant-source preserve re-expresses the GGUF blocks that are already there as MLX affine groups (w = scale * code + bias) on the 32-value group grid the blocks use, instead of decoding to FP16 and quantizing again. The integer codes are kept; only the fp16 rounding of the scale and bias is new. Measured against gguf.quants.dequantize, per-tensor relative L2 is 2.0-2.4e-4 for Q4_K, 0.0 (bit-exact) for Q8_0, and 2.2e-4 for Q6_K when a 32-value group's two 16-value sub-block scales agree.

Q6_K has a structural limit: its sub-block scales are 16 values wide, so when they disagree inside a 32-value group it cannot be represented exactly. Those tensors are flagged exact: false and kept in FP16 rather than silently requantized, and conversion_report.json records the decision plus per-tensor max abs error, relative L2 and an aggregate 1e-3 gate. Modules end up with mixed 4/6/8-bit widths, which mlx-lm loads via per-module quantization entries.

Preserve fails closed: llama, qwen2, qwen3 and mistral are supported, and any other architecture errors out naming the arch instead of falling back to requantization. --preserve-source-quant still works as a deprecated alias.

gguf2mlx convert --input model-Q8_0.gguf --output ./mlx-model-q8 --quant-source preserve

Publish to the Hugging Face Hub

gguf2mlx publish downloads a hosted GGUF, converts it, and uploads the result with a provenance record and the source license files. Split GGUFs are merged; a partial shard set is rejected before anything is downloaded, and the destination repository is only created once conversion produced weights.

pip install "gguf2mlx[publish]"
gguf2mlx publish --source bartowski/Model-GGUF --gguf Model-Q8_0.gguf \
  --dest my-org/Model-MLX --source-revision main
# add --quantize --q-bits 4 --q-group-size 64 for affine 4-bit (needs MLX)

The reusable publish-mlx.yml workflow is a thin shell around the same module: float16 runs on ubuntu-latest, and only quantize: true needs a macOS Apple Silicon runner. Details: Publishing.

Supported models

Conversion is adapter-based, not "any GGUF". Rows marked ⚠️ convert but still need opt-in real-checkpoint validation.

preserve in the Convert column means --quant-source preserve is available (Q4_K, Q6_K, Q8_0). Other architectures convert, but must requantize.

Inspection-only (recognized via metadata, rejected on conversion): arctic, bert, bitnet, bloom, dbrx, falcon, gpt2, gptneox, mpt, olmo, phi, phi2, t5 and others — full list in the support matrix.

Vision-language models

Three explicit adapters emit directories for mlx-vlm: LLaVA-1.5, Qwen2-VL, and MiniCPM-V 4.7 1B. Language weights and the projector come from GGUF; hybrid mode takes only the vision tower from a matching HF checkpoint. Qwen2.5-VL, Qwen3-VL full VLM, LLaVA-NeXT and SigLIP layouts are not supported by these adapters.

pip install "gguf2mlx[vlm]"
gguf2mlx convert --type vlm \
  --input qwen2-vl.gguf --mmproj qwen2-vl-mmproj.gguf \
  --hf-model Qwen/Qwen2-VL-2B-Instruct --output ./qwen2-vl-mlx

Flags, hybrid/offline paths, quantization policy and the Q4_K --preserve-source-quant mode: VLM conversion. Native acceptance results: Qwen2-VL report.

Qwen3-VL text-only extraction is a separate --type text-encoder contract (BF16 by default, mRoPE preserved, vision tensors skipped): see text conditioning.

Limits worth knowing before you start

• --quantize is a second quantization. Use --quant-source preserve to keep the original blocks; for Q6_K tensors whose sub-block scales disagree, that still means FP16.
• Conversion cannot restore precision lost in the source GGUF.
• Tokenizer fidelity depends on GGUF metadata; embedded HF tokenizer JSON is preserved when present, otherwise reconstructed from the header.
• Mixed-precision MoE output is experimental and needs a JANG-compatible runtime; use uniform output when interoperability is unverified.
• No universal speed claim versus llama.cpp — measure on your hardware.

Documentation

Quality gates · Support matrix · How conversion works · VLM conversion · Text conditioning · Publishing · .mlx bundles · Capabilities · Benchmarks · FAQ · Roadmap · Validation logs · Development

Development

git clone https://github.com/barrontang/gguf2mlx.git && cd gguf2mlx
uv sync --all-extras
pytest -q                                   # portable suite
GGUF2MLX_RUN_E2E=1 pytest tests/test_e2e.py # Apple Silicon, real GGUF
ruff check src/ tests/ benchmarks/

CI runs Python 3.10/3.12/3.13 with an 80% coverage floor; a separate macOS workflow runs the native VLM suites. Details and opt-in fixture paths are in Development.

Contributing

New architecture adapters must ship with a tensor manifest and a load test. When requesting a family, include the exact GGUF architecture, model name, quantization type, tokenizer type, and a public fixture URL.

License

Apache-2.0. The traffic-tracking dashboard that used to live here was GPL-3.0 and now lives in its own repository.