mlx-model-doctor

Validate an MLX / Hugging Face model repository before you load it.

A model repo can be broken in ways you only discover halfway through load(): a config.json that's missing or internally inconsistent, a missing tokenizer file, a model.safetensors.index.json that points at shards that aren't there, quantization metadata that uses a mode or group size MLX rejects, a chat template that's absent or whose stop token has a typo, a corrupt safetensors header, a quantized layer whose packed weight and scales shapes disagree, or a model that simply won't fit in the memory you have. mlx-model-doctor checks those up front and prints a report, so a bad repo fails fast with a clear reason instead of a confusing crash.

The checks read repository metadata and the safetensors header — config.json, the tokenizer files, the safetensors index, quantization fields, and the tensor map (dtypes, shapes, byte-offsets) parsed from the header alone. They need no GPU or MLX and never download the weights; on the Hub the header arrives over a small HTTP range request, so the checks stay cheap to run anywhere. For the default text profile, an optional --smoke check loads the model through mlx-lm (Apple Silicon) under a memory cap, to confirm it loads and generates. VLM smoke through mlx-vlm is future work.

mlx-model-doctor check local ./my-model
mlx-model-doctor check hf mlx-community/Llama-3.2-3B-Instruct-4bit

See EXAMPLES.md for real, dated transcripts plus newer command examples.

Install

pip install mlx-model-doctor
# With the optional mlx-lm smoke check (Apple Silicon):
pip install "mlx-model-doctor[mlx-lm]"

Or with uv:

uv add mlx-model-doctor
uv add "mlx-model-doctor[mlx-lm]"

Verify the install:

mlx-model-doctor version
mlx-model-doctor --help

Requires Python ≥ 3.11. The static checks are pure Python and need only huggingface-hub; the optional --smoke runtime check currently applies to the default text profile and needs mlx-lm plus Apple Silicon.

What it checks

The built-in text plugin runs these against a model repository, broadly in this order:

text is the default plugin. For vision-language repositories, pass --plugin vlm to run the explicit non-runtime VLM profile. The VLM plugin keeps the same metadata and safetensors-header posture: no weights are downloaded and no model runtime is loaded.

• Required files — config.json is present and readable.
• Config consistency — config.json parses, and its model_type is set.
• Tokenizer — the tokenizer files a text model needs are present, and the special-token configuration is coherent.
• Chat template — a chat/instruct model declares a chat template (in tokenizer_config.json, chat_template.json, or a chat_template.jinja), and the end-of-turn token its template emits is a registered special token. A typo'd stop token loads fine and then never stops generating.
• Safetensors index — when the weights are sharded, model.safetensors.index.json is valid and every shard it references exists.
• Tensor header — read the safetensors header itself (the tensor map: dtypes, shapes, byte-offsets; no weight download) to catch a corrupt header (overlapping or out-of-bounds tensor offsets), a weight map that points at tensors no shard contains, a declared tied embedding that contradicts the stored weights, and an MLX-quantized layer whose packed-weight and scales shapes don't agree. These run by default; pass --skip-weights to skip them for a faster config-only pass.
• Quantization metadata — quantization fields are present and use a valid MLX mode with a valid group size and bit width (affine, mxfp4, mxfp8, nvfp4). This reads the metadata, not the tensors.
• Generation tokens — the eos / pad / bos token IDs are present and agree across config.json, generation_config.json, and tokenizer_config.json.
• Memory budget — an estimate of the memory the model needs at your context length, compared against a budget you pass with --max-memory.

Each check returns a result with a status (pass / warn / fail / skip), a message, and — when something is wrong — a remediation hint. The report aggregates them, and the process exit code reflects the worst result under your fail policy.

Python API

from mlx_model_doctor import check_local_model, check_hf_model

report = check_local_model("./my-model")
print(report.summary)            # {"pass": 9, "warn": 1, "fail": 0, "skip": 2}
for result in report.results:
    print(result.status, result.check_id, result.message)

# Hugging Face repos (hits the Hub):
report = check_hf_model("mlx-community/Llama-3.2-3B-Instruct-4bit")

DoctorReport renders to text, JSON, or Markdown (render_text / render_json / render_markdown), and the result objects are frozen dataclasses, so the output is stable to diff in CI.

Commands

check accepts --format {text,json,markdown,github}, --output <file>, --max-memory <e.g. 32gb>, --context-length <n>, --fail-on {error,warn,never}, --skip-weights (skip the tensor-header checks for a faster config-only pass), and --smoke for the default text smoke backend. Use --plugin vlm for vision-language repositories; omit it for the default text profile. The github format prints GitHub Actions annotations (see Use it in CI).

Exit codes: 0 checks passed (under the fail policy), 1 checks found failures, 2 tool error — a bad target, a missing dependency, or zero checks run.

The Hugging Face path

check hf and sample hf talk to the Hub through huggingface-hub. They read repository metadata (the file list, sizes, the small text files, and the safetensors header over a range request) rather than downloading the weights, but they do need network access, and an auth or rate-limit problem surfaces as a clear tool error rather than a stack trace. sample hf is a survey: it lists an author's repos, keeps the ones that look like MLX models, validates a deterministic sample of them, and reports each as its own batch item — a per-model failure is recorded and the run continues.

Validate before uploading to Hugging Face

For a model you build or convert locally, run the static checks before upload and treat warnings as release blockers unless you have reviewed them:

mlx-model-doctor check local ./dist/my-mlx-model --fail-on warn
hf upload my-org/my-mlx-model ./dist/my-mlx-model --repo-type model
mlx-model-doctor check hf my-org/my-mlx-model --fail-on warn

If your publisher is Python-based, keep the same order:

from huggingface_hub import upload_folder

# Run `mlx-model-doctor check local ./dist/my-mlx-model --fail-on warn` first.
upload_folder(
    repo_id="my-org/my-mlx-model",
    folder_path="./dist/my-mlx-model",
    repo_type="model",
)

Use --fail-on warn before upload when you want a clean producer release gate. Use the default --fail-on error when warnings are acceptable but hard failures should still block. After upload, check hf verifies that the Hub repository exposes the same files and metadata the local directory did.

Check a fused adapter kept its behavior

Fusing a LoRA adapter into a base model is supposed to bake the adapter's learned behavior into the weights. It doesn't always. With mlx-lm's default fuse the merged weights get re-quantized back to the base's format, and that round-trip can erase a good part of what the adapter learned — so the fused model quietly behaves more like the plain base than like the model you fine-tuned. parity mlx catches that before you ship the fused model.

mlx-model-doctor parity mlx \
  --base ./Qwen2.5-0.5B-Instruct-4bit \
  --adapter ./my-lora \
  --fused ./my-fused-model \
  --prompts ./examples.json

It runs the same teacher-forced input through three models — the base alone, the base with the adapter loaded, and the fused model — and compares their top-token predictions. If the fused model tracks base+adapter the verdict is pass; if it fell back to the base and lost the adapter, fail_tracks_base; if it matches neither, fail_gross; and if the difference is within measurement noise, inconclusive. A default 4-bit fuse that only kept half the adapter lands at inconclusive — parity not confirmed — rather than a misleading pass. The exit code is 0 for a confirmed pass, 1 for a determined regression, and 2 when the run can't decide (inconclusive or a setup problem).

--prompts takes a JSON list of {"prompt", "completion"} examples and builds the comparison fixture from them, tokenized with the base model's own tokenizer and scored over the completion tokens — so you measure parity on text that matters for your model. Use examples the adapter should have changed. The check needs the optional [mlx-lm] extra, a base repository that ships a tokenizer.json, and loads each model in its own memory-capped subprocess (one at a time), so it's safe on a laptop. If your fuse comes out inconclusive or worse, re-fuse with --dequantize (a float fuse), which preserves the adapter.

Point --base, --adapter, and --fused at model directories of real files — a locally converted or fused model, or one fetched with hf download <repo> --local-dir ./dir. The raw Hugging Face cache path (~/.cache/huggingface/hub/models--.../snapshots/...) stores its files as symlinks into a sibling blobs/ directory, which the checker does not yet follow.

parity mlx also accepts --format {text,json,markdown}; the json output follows the published parity.v1 schema.

Use it in CI

Gate a pull request on a model repository with the GitHub Marketplace Action. It runs the static checks (no weights downloaded, no GPU), writes the report to the job summary, and fails the job under your fail policy:

- uses: IonDen/mlx-model-doctor@v0
  with:
    source: hf
    target: mlx-community/Llama-3.2-3B-Instruct-4bit
    plugin: text
    fail-on: warn

For a vision-language model, opt into the VLM profile:

- uses: IonDen/mlx-model-doctor@v0
  with:
    source: hf
    target: mlx-community/InternVL3-2B-4bit
    plugin: vlm

Add version: "==0.7.0" to pin the tool to a release; without it the action installs the latest published version.

For a model directory you keep in git, validate it on every commit with the pre-commit hook:

repos:
  - repo: https://github.com/IonDen/mlx-model-doctor
    rev: v0.7.0
    hooks:
      - id: mlx-model-doctor
        args: ["path/to/model"]

Output contract

--format json prints a stable, versioned payload. The top-level fields are:

Each result in results[] has:

The machine-readable schema ships with the package at mlx_model_doctor/schema/report.v1.schema.json and is validated against real output in CI.

Exit codes: 0 checks passed under the fail policy, 1 failures found, 2 a tool error (a bad target, a missing dependency, or zero checks run). --format github reports the same results as GitHub Actions annotations; inside a workflow it also writes the Markdown report to the job summary and the pass / warn / fail / skip / exit-code / schema-version values to the step outputs.

Stability policy

Public API

The names you can depend on — only change on a major release:

check_local_model, check_hf_model, CheckOptions, DoctorReport, CheckResult, render_json, render_text, render_markdown, render_github, exit_code_for, FailOn, and the error types ModelDoctorError, TargetError, DependencyError, MemorySafetyError. exit_code_for raises ValueError on an unrecognized fail-on value.

Internal layer

The check, plugin, and target Protocols; CheckContext; the plugin registry; and the hub= parameter on check_hf_model (a test injection seam whose type may change) are internal and not stable across releases.

Schema versioning

schema_version is MAJOR.MINOR, versioned independently of the package. A minor bump adds new optional fields or new values to open fields (such as the memory check's estimate_source values). A major bump means a documented field was removed, renamed, or retyped, or a closed enum (status, severity, source) changed.

The top-level object, summary, and each entry in results[] are closed (additionalProperties: false), so a new field there is a coordinated schema edit plus a minor version bump. Validate against the schema that matches the payload's schema_version, not a pinned older copy — otherwise a newer payload's added field will fail your validator.

Promoted details keys

details is otherwise free-form, but three keys from the memory check are stable across the 1.x schema line: lower_bound_bytes, estimate_source, and memory_lower_bound_kind. lower_bound_bytes is a structural lower bound — it counts attention, MLP, and embedding parameters at ≤16-bit weights (or quantized-equivalent) plus KV cache, but excludes norms, biases, and an untied lm_head. It sits below real runtime use and is not a fit guarantee.

Batch output

The sample hf --format json survey has its own published schema, at mlx_model_doctor/schema/sample-batch.v1.schema.json and validated against real output in CI. It carries a schema_version of sample-batch/MAJOR.MINOR (same bump rules as above), and each checked item embeds a full single-check report that conforms to report.v1.schema.json.

Version sensitivity

The checks encode behavior from specific upstream versions (MLX, transformers, safetensors). When a value falls outside the known set — a quantization mode added in a newer MLX, an unfamiliar safetensors dtype — the check warns rather than failing. Only structurally invalid metadata (wrong type, missing required fields) produces a failure. Each version-bound table carries a comment naming the upstream version it was verified against.

Status

Beta (0.9.1). Version 0.9.1 is a false-positive cleanup: four checks were flagging valid, popular repositories and now pass — a safetensors index kept in a subdirectory (read as a "missing shard"), a model with tie_word_embeddings absent and no separate output head (Gemma), a Pixtral/Mistral-3 [IMG] image token, and a vision-language repository whose chat template lives in chat_template.json. Genuine problems still fail. The static check local path and the report/CLI surface are solid and well tested. Version 0.9.0 added the adapter-parity verifier (parity mlx / check_adapter_parity), which checks whether fusing a LoRA adapter into a base model kept the adapter's behavior — see Check a fused adapter kept its behavior; it needs the optional [mlx-lm] extra and a base repository that ships a tokenizer.json. Version-bound check tables (MLX quantization modes, safetensors dtypes) now warn rather than fail on a recognized-but-unlisted value, so a repository built against a newer upstream release is flagged as unverified instead of rejected outright — see Version sensitivity. The optional runtime smoke check now covers vision-language repositories too: --smoke --plugin vlm loads a model through mlx-vlm and generates from a dummy image under the same advisory memory caps as the text path, with remote code execution refused by default. sample hf gained a configurable scan depth (--max-candidates), a listing-signal filter (--signal-filter), and a local listing cache (--no-cache / --cache-ttl) for surveying more of an author's catalog without repeating Hub calls. The safetensors header (read without downloading weights) backs four tensor-level checks — offset corruption, weight-map parameter sanity, tied-embedding consistency, and MLX quantized-layer shape consistency — which run by default (--skip-weights opts out). A single check reports whether a repository looks like an MLX model and why; the VLM profile adds image-processor and image-token wiring checks for vision-language repositories. The quantized-shape and quantization-mode checks read each layer's own bits/group_size/mode, so a mixed-precision model (4-bit experts with 8-bit dense and router layers) is validated per layer rather than reported as broken. The memory estimate handles mixed precision the same way: when a model mixes bit widths it takes the weight figure from the stored file sizes instead of the model-level setting. The Hugging Face path (check hf, sample hf) is implemented and tested offline against fakes; its live behavior is exercised by opt-in network tests. It also ships a GitHub Action and a pre-commit hook. The public API and JSON output now have a documented, versioned stability contract — see Output contract and Stability policy. Pin a version if you depend on the schema or the API.

License

Apache-2.0. See LICENSE and NOTICE. Validating a model repository does not touch the model's own weights or license; those belong to their respective authors.

Acknowledgements

• Apple ML Explore for MLX and mlx-lm.
• Hugging Face for the Hub client and safetensors.

Sister projects

Other MLX libraries for Apple Silicon:

• mlx-taef — tiny autoencoders for fast diffusion-latent previews and low-memory decode (FLUX / SD).
• mlx-teacache — TeaCache residual caching to skip redundant FLUX denoising steps.
• mlx-quant-fidelity — measure how much quality a quantization costs (KL divergence, top-token flips, perplexity delta).

By Denis Ineshin · ineshin.space