quantpreflight 0.4.0

Find out if a quantized AI model will run on your GPUs, before you download it.

Try it in your browser  ·  Quick start  ·  What it checks  ·  Accuracy  ·  Docs

A quantized checkpoint can be 40 to 400 GB. You pull it onto your GPU nodes, start the engine, and only then learn that your GPU has no kernel for its format, or that its tensors won't split across your GPUs, or that it doesn't fit. quantpreflight tells you first, from a few hundred KiB of config files and tensor headers.

Highlights

• Answers in seconds. It reads headers, not weights: 887 KiB for a 38.74 GiB checkpoint.
• Needs no GPU. There's no PyTorch and no CUDA, and no dependencies at all on Python 3.11+. It runs on a laptop or in CI.
• Checks the whole mix. 16 rules cover the model, format, engine, GPU, and GPU count together, because that's where most failures come from.
• Tells you what works. It gives you the GPU count and context length that start, with the vllm serve command to run. suggest finds the quantized versions of a model that run on your GPUs.
• Tested on real hardware. It predicted 26 of 27 real vLLM loads on Tesla T4, and it has scanned 505 public models with no crashes. See the results.
• Finds known vulnerabilities. --engine-version checks your vLLM or SGLang version against published advisories.
• Fits your workflow. Use the CLI, a GitHub Action, a pre-commit hook, SARIF output, or the browser scanner.

Quick start

Install it from PyPI. It needs Python 3.10 or newer:

pip install quantpreflight

Then name a model and where you plan to run it:

quantpreflight check nvidia/Qwen3-8B-FP4 --gpu H100

quantpreflight v0.4.0
  Source:       nvidia/Qwen3-8B-FP4
  Architecture: Qwen3ForCausalLM (safetensors)
  Quantization: nvfp4 4-bit g16 nvfp4-pack-quantized
  Weights:      5.96 GiB across 2 file(s), 1227 tensors
  Headers read: 301.0 KiB (zero weight bytes downloaded)
  Target:       engine=vllm, gpu=H100, tp=1
  Est. VRAM:    8.24 GiB / GPU (weights, runtime, and KV cache)

Findings: 1 (1 critical)
------------------------------------------------------------------------
[1] CRITICAL QPF004: NVFP4 requires Blackwell (sm_100+), target is H100 (sm_90)
    Location: config.json
    Detail:   Checkpoint uses NVFP4 4-bit floating-point quantization, which is not supported on H100 (sm_90).
    Fix:      Deploy NVFP4 checkpoints on Blackwell GPUs (B200/GB10/RTX5090) or use AWQ/GPTQ/FP8.

Setup: no --tp or context length makes this start.
  QPF004 needs a different checkpoint, GPU, or engine.
  Find one that works: quantpreflight suggest nvidia/Qwen3-8B-FP4 --gpu H100

So find one that works. suggest scans the quantized versions of the model on Hugging Face and ranks the ones that start:

quantpreflight suggest nvidia/Qwen3-8B-FP4 --gpu H100 --limit 4

quantpreflight v0.4.0
Versions of Qwen/Qwen3-8B for vLLM on H100, tp=1:

  Setup    Model                              Format                                Weights    Details
  ready    Qwen/Qwen3-8B                      unquantized (BF16)                    15.26 GiB  tp=1, 40960 tokens
  ready    Qwen/Qwen3-8B-FP8                  fp8 8-bit block-fp8                   8.79 GiB   tp=1, 40960 tokens
  ready    Qwen/Qwen3-8B-AWQ                  awq 4-bit g128 gemm                   5.68 GiB   tp=1, 40960 tokens
  blocked  nvidia/Qwen3-8B-NVFP4              nvfp4 4-bit g16 nvfp4-pack-quantized  5.96 GiB   QPF004
  blocked  unsloth/Qwen3-8B-unsloth-bnb-4bit  bitsandbytes 4-bit nf4                6.97 GiB   QPF011

Best pick: Qwen/Qwen3-8B, the highest precision that starts.
  Run:  vllm serve Qwen/Qwen3-8B

When a setting is all that's wrong, check says which one to change. Here a 32B model is too big for one L4:

Setup: it starts with these changes.
  - Use --tp 2 instead of 1, so the weights fit in memory.
  Fits 32768 tokens per request at tp=2.
  Run:  vllm serve Qwen/Qwen2.5-32B-Instruct-AWQ --tensor-parallel-size 2

On the machine you'll serve from, --gpu auto reads its GPUs and --engine-version auto reads the installed vLLM or SGLang.

It takes a Hugging Face repo ID, a direct URL, a local folder, or a single .safetensors or .gguf file:

quantpreflight check Qwen/Qwen2.5-7B-Instruct-AWQ --engine vllm --gpu H100 --tp 4
quantpreflight check ./models/deepseek-r1-fp8 --gpu L40S --tp 8
quantpreflight check ./models/llama-3-8b-q4_k_m.gguf --engine llamacpp --gpu L4

Want to see it find problems? This repository has a deliberately broken sample that holds headers only:

git clone https://github.com/layer8labs8/quantpreflight.git
quantpreflight check quantpreflight/tests/fixtures/broken-model --gpu H100 --tp 4

The usage guide covers every option. Set HF_TOKEN to scan gated models.

What it checks

Run quantpreflight explain QPF006 for the details of any rule.

Supported

Use it in CI

- uses: layer8labs8/quantpreflight@v0.4.0
  with:
    target: ./exported-checkpoint
    engine: vllm
    gpu: H100
    tp: "4"
    fail-on: high

The usage guide shows SARIF upload to the Security tab and a pre-commit hook.

How accurate is it?

Every result here can be reproduced from this repository. The accuracy report has the full tables.

We wrote the benchmark ourselves, so treat it as a regression check. Only T4 GPUs have been tested so far.

How it fits with other tools

Run it before the download. These tools do different jobs:

What it can't do

A scanner finds known problems, not proof. A clean report doesn't mean the model will load or answer well, and a finding may not apply to your exact engine version.

• It never reads weight values, so it can't see NaNs or measure quality loss.
• Its GPU rules were checked on T4 only. Other GPUs rest on vendor docs and engine issues until someone tests them.
• Its compatibility tables can fall behind a new engine release.
• The memory estimate assumes one request at a time unless you pass --concurrency. Its overhead was fitted on T4.
• It only checks your engine version if you pass --engine-version, and only vLLM and SGLang have advisory data.
• The browser scanner reads public Safetensors models or a pasted config.json. GGUF, gated models, and local folders need the command line.

FAQ

Does it download the model? No. It reads config files plus the header at the start of each weight file. A scan never fetches weight data.

Do I need a GPU or PyTorch? No. Pass the GPU you plan to use with --gpu, and it checks against a built-in table.

It flagged a model that runs fine. Is that a bug? It might be. Please open a report. False alarms are the most useful feedback we get.

Is my token safe? HF_TOKEN goes only to Hugging Face, and it's removed if a download redirects to another host. See SECURITY.md.

Help test it on your GPU

Only Tesla T4 has been checked against real loads so far. If you can run vLLM on another GPU, one run of tools/gpu_check.py shows where the rules are right and where they're wrong. GPU testing explains how, and the help wanted issues list the GPUs we need most.

Contributing

Most contributions are one small TOML file. Adding a model family or a GPU takes no Python and no GPU. Start from an existing file in architectures/ or hardware/, and see CONTRIBUTING.md.

Support

If this project helped you, I'm really glad. If you'd like to support it, you can send a crypto tip. If not, that's completely okay.

License

MIT