quantpreflight 0.3.0

Find out if a quantized AI model will run on your GPUs, before you download it.

Try it in your browser  ·  Quick start  ·  What it checks  ·  Accuracy  ·  Docs

A quantized checkpoint can be 40 to 400 GB. You pull it onto your GPU nodes, start the engine, and only then learn that your GPU has no kernel for its format, or that its tensors won't split across your GPUs, or that it doesn't fit. quantpreflight tells you first, from a few hundred KiB of config files and tensor headers.

Highlights

• Answers in seconds. It reads headers, not weights: 887 KiB for a 38.74 GiB checkpoint.
• Needs no GPU. There's no PyTorch and no CUDA, and no dependencies at all on Python 3.11+. It runs on a laptop or in CI.
• Checks the whole mix. 16 rules cover the model, format, engine, GPU, and GPU count together, because that's where most failures come from.
• Tested on real hardware. It predicted 26 of 27 real vLLM loads on Tesla T4, and it has scanned 505 public models with no crashes. See the results.
• Finds known vulnerabilities. --engine-version checks your vLLM or SGLang version against published advisories.
• Fits your workflow. Use the CLI, a GitHub Action, a pre-commit hook, SARIF output, or the browser scanner.

Quick start

Install it from PyPI. It needs Python 3.10 or newer:

pip install quantpreflight

Then name a model and where you plan to run it:

quantpreflight check nvidia/Qwen3-8B-FP4 --gpu H100

quantpreflight v0.3.0
  Source:       nvidia/Qwen3-8B-FP4
  Architecture: Qwen3ForCausalLM (safetensors)
  Quantization: nvfp4 4-bit g16 nvfp4-pack-quantized
  Weights:      5.96 GiB across 2 file(s), 1227 tensors
  Headers read: 301.0 KiB (zero weight bytes downloaded)
  Target:       engine=vllm, gpu=H100, tp=1
  Est. VRAM:    8.24 GiB / GPU (weights, runtime, and KV cache)

Findings: 1 (1 critical)
------------------------------------------------------------------------
[1] CRITICAL QPF004: NVFP4 requires Blackwell (sm_100+), target is H100 (sm_90)
    Location: config.json
    Detail:   Checkpoint uses NVFP4 4-bit floating-point quantization, which is not supported on H100 (sm_90).
    Fix:      Deploy NVFP4 checkpoints on Blackwell GPUs (B200/GB10/RTX5090) or use AWQ/GPTQ/FP8.

It takes a Hugging Face repo ID, a direct URL, a local folder, or a single .safetensors or .gguf file:

quantpreflight check Qwen/Qwen2.5-7B-Instruct-AWQ --engine vllm --gpu H100 --tp 4
quantpreflight check ./models/deepseek-r1-fp8 --gpu L40S --tp 8
quantpreflight check ./models/llama-3-8b-q4_k_m.gguf --engine llamacpp --gpu L4

Want to see it find problems? This repository has a deliberately broken sample that holds headers only:

git clone https://github.com/layer8labs8/quantpreflight.git
quantpreflight check quantpreflight/tests/fixtures/broken-model --gpu H100 --tp 4

The usage guide covers every option. Set HF_TOKEN to scan gated models.

What it checks

Run quantpreflight explain QPF006 for the details of any rule.

Supported

Use it in CI

- uses: layer8labs8/quantpreflight@v0.3.0
  with:
    target: ./exported-checkpoint
    engine: vllm
    gpu: H100
    tp: "4"
    fail-on: high

The usage guide shows SARIF upload to the Security tab and a pre-commit hook.

How accurate is it?

Every result here can be reproduced from this repository. The accuracy report has the full tables.

We wrote the benchmark ourselves, so treat it as a regression check. Only T4 GPUs have been tested so far.

How it fits with other tools

Run it before the download. These tools do different jobs:

What it can't do

A scanner finds known problems, not proof. A clean report doesn't mean the model will load or answer well, and a finding may not apply to your exact engine version.

• It never reads weight values, so it can't see NaNs or measure quality loss.
• Its GPU rules were checked on T4 only. Other GPUs rest on vendor docs and engine issues until someone tests them.
• Its compatibility tables can fall behind a new engine release.
• The memory estimate assumes one request at a time unless you pass --concurrency. Its overhead was fitted on T4.
• It only checks your engine version if you pass --engine-version, and only vLLM and SGLang have advisory data.
• The browser scanner reads public Safetensors models or a pasted config.json. GGUF, gated models, and local folders need the command line.

FAQ

Does it download the model? No. It reads config files plus the header at the start of each weight file. A scan never fetches weight data.

Do I need a GPU or PyTorch? No. Pass the GPU you plan to use with --gpu, and it checks against a built-in table.

It flagged a model that runs fine. Is that a bug? It might be. Please open a report. False alarms are the most useful feedback we get.

Is my token safe? HF_TOKEN goes only to Hugging Face, and it's removed if a download redirects to another host. See SECURITY.md.

Help test it on your GPU

Only Tesla T4 has been checked against real loads so far. If you can run vLLM on another GPU, one run of tools/gpu_check.py shows where the rules are right and where they're wrong. GPU testing explains how, and the help wanted issues list the GPUs we need most.

Contributing

Most contributions are one small TOML file. Adding a model family or a GPU takes no Python and no GPU. Start from an existing file in architectures/ or hardware/, and see CONTRIBUTING.md.

Support

If this project helped you, I'm really glad. If you'd like to support it, you can send a crypto tip. If not, that's completely okay.

License

MIT