quantprobe

the bar is the probe: one column through the memory tiers it prices — VRAM, RAM, disk.

Predicts how fast an LLM will run on your machine — before you download it — then hands you the exact llama.cpp command. If nothing fits well enough, it builds a quantization tuned to your specific model and hardware. Then it proves the result: speed re-measured on demand, quality scored by KL divergence, and the recommended config put through 40 machine-checked business tasks before we called it usable.

Quickstart · Browser version · What runs on what · Commands · The laws · When it won't help

Validated, not vibes. 14-model ladder at 8.4% median error on measured hardware · every printed all-in-VRAM number a documented floor (real speed ≥0.90× on 13/13 benchmarks, typically 1.1–1.8× higher) · retrodicts third-party results it never trained on (airllm's 30× spread, DGX Spark reports, a 1.56 TB Kimi rig) · every prediction staked before measuring, and the misses published at the same size as the hits.

Quickstart

pip install quantprobe
quantprobe plan --model qwen3-30b

[quantprobe] no hardware flags: auto-detected this machine (vram 6GB@192 | ram 16GB@48 | disk 0.5 GB/s).
[quantprobe] calibration applied [ram 24.3 GB/s measured; disk 3.13 GB/s measured] (2026-07-28)
[quantprobe] anchored: CPU x1.18, GPU x0.75 from your calibrate anchor runs [tier ratios; --no-anchors disables]

quantprobe plan - Qwen3-30B-A3B @ 2.5-bit on THIS machine [auto-detected]
  model 10.6 GB | active 1.53 GB/token | est. quality cost x1.07 (depth-aware recipe)

  *   22.2 tok/s  split experts: 34%->VRAM, rest->RAM   [pins 7GB of 12GB RAM (CUDA host memory) - …]
      19.0 tok/s  hybrid: attention->VRAM, experts->RAM   [pins 10GB of 12GB RAM (CUDA host memory) - …]
      13.2 tok/s  pure CPU (GPU idle)   [RAM boundary - expect bimodal speed]

  binding constraint: BANDWIDTH-BOUND (system RAM bandwidth) - 51% of every decode token is spent there.

  run it:  llama-server -m model.gguf -ngl 99 -ot "blk\.(16|17|…|47)\.ffn_.*_exps\.=CPU" --no-mmap -b 1024 -ub 1024 --threads 4

The first line is what a fresh install prints. The calibration applied and anchored: lines appear after you run quantprobe calibrate once — measured constants and your own anchor runs, not spec sheets. The binding constraint line is the part most tools never tell you: 3 tok/s, disk-bound means buy RAM; 3 tok/s, bandwidth-bound means don't bother.

Downloads nothing. Takes a second. No hardware flags needed — it reads your machine. --model and --bits just say what you're considering; point it at a file you already have with --gguf model.gguf instead.

Want it to do the whole thing for you?

quantprobe auto

Detects your machine, asks which model you want, picks the best quant for it, downloads it, and launches. No flags to learn.

What it does

One pipeline, end to end — most tools ship the first line only:

• Predicts tok/s before you download — across every placement (all-VRAM, hybrid, expert-split, CPU, disk-stream) and picks the winner, printing which resource binds and what fixing it would buy.
• Anchors predictions to your machine — run quantprobe calibrate once and two benchmark runs on your own GGUF scale every prediction. That path passed the gate it pre-registered before any number existed (prereg #64): leave-one-out median error 19% → 5.8% across 5 arms; ~12% median on the full ladder, misses erring low. --no-anchors restores the plain law.
• Emits the exact command, including the -ot regex most guides get wrong.
• Finds free speed in what you already have — partial expert offload and prompt-lookup speculation need no new download.
• Builds layer-aware quantizations — measures which layers of your model break under compression, then protects them (−9% perplexity at the same file size).
• Audits a running Ollama install — quantprobe audit-ollama reads the placement Ollama actually chose, prices it against the planner's, and refuses to compare while VRAM is contended (a measurement discipline most benchmarks skip).
• Proves quality, not just speed — perplexity and full-distribution KL divergence via llama.cpp's own --kl-divergence, because we measured perplexity moving 23% while the model changed its chosen token on 27% of positions.
• Tells you when to stop — it declines the expensive path on machines that don't need it.
• Runs on stock llama.cpp. No custom runtime, nothing to build.

Does the cheap quant actually do the work?

Speed numbers are worthless if the model can't do the job. So we staked a bar before generating a single output — ≥80% of machine-checked tasks or the config is business-useful; under 60% and every tok/s figure we publish gets qualified — and ran the recommended 2.5-bit 30B through 40 auto-scored business tasks: JSON extraction with exact values, arithmetic to the cent, single-label classification, code that must execute and pass assertions, summaries where any number not present in the source fails a deterministic hallucination check.

Result: 40/40. Five tasks initially exhausted a 4k context window mid-reasoning; at 16k all five pass (one needed 7,417 tokens of thinking — reasoning models spend their budget before they answer). Honest floor if you count those five as failures anyway: 85%, still above the staked bar. Full outputs, every check, every verdict →

The task set also carries a difficulty ladder for comparing models on identical predicates — up to a tier designed so today's models fail it:

* thinking-model truncations quarantined and disclosed, never counted as failures. The 0.6B fires the suite's own kill rule (57.9% < 60%) — the instrument correctly refuses to call it business-usable. And one honest anomaly the ladder itself exposed: the only T4 task anyone solved (the 5-house logic puzzle) was solved by the biggest and the smallest model while both 7Bs failed it — non-monotonic in capability, the signature of training-data recall rather than reasoning. That task is being replaced with a generated-novel variant; the score stands as recorded.

Every T3/T4 answer key is recomputed mechanically by the self-test and both logic puzzles are brute-forced to exactly one solution before any model is scored. The T4 nine-digit multiplication is the tier working as intended: the model announced it would need a calculator, then printed a confident 18-digit answer that is wrong at digit 5.

Check any speed claim without owning the hardware

Law 4 is tok/s = η·BW ÷ bytes-per-token, and it prices other people's machines as well as yours (the full hardware × model matrix →):

• "DGX Spark runs 70B Q4 at 35–45 tok/s" — a 70B dense at Q4 moves 42.5 GB per token; at 273 GB/s the perfect-efficiency ceiling is 6.4 tok/s. The claim needs 5.5–7× the bandwidth the hardware has. Whatever was measured, it wasn't single-stream decode.
• A 1.56 TB Kimi K3 rig reported as "10 tok/s" — the repo's own README says seconds per token; the relay inverted the unit by 200–320×. Better: its four RAM presets test the law. "Add RAM" predicts 15.6× speedup; Law 4 predicts almost none (the expert working set can't be cached); measured across the presets: 1.63×.
• airllm's unexplained 30× spread (0.07–2 tok/s across hosts) — the law retrodicts it as a tier boundary: RAM-resident hosts land on the RAM term, disk-bound hosts on the disk term.

Same arithmetic the planner runs — you just feed it someone else's bandwidth and bytes.

One box, two right answers — it depends how many people are using it

At one user the 30B MoE is the better model — smarter, and 19.7 tok/s. At 32 users the dense 7B wins by 5.5× on aggregate throughput, because routed-expert reads from system RAM do not amortise across streams while dense weights read once serve everyone. The jump at width 8→9 is a kernel switch, not a smooth curve — which also makes batch widths 2–8 strictly dominated on this card class. plan prints the right advice for whichever placement it recommends (U-38 overturned our own prior "2× ceiling"; U-39 confirmed the MoE cap as staked).

Fast vs Custom

Most people want Fast. Above ~3 bits per weight, community quants are already near-lossless — so --custom refuses to run on machines that don't need it and says why. Reach for Custom when you're squeezing a model that barely fits (under ~3 bits, where ordinary compression falls off a cliff), when you have a fine-tune nobody has published, or when you need maximum quality at a fixed size.

Free speed you probably already have

Most guides put all of a mixture-of-experts model's experts in system RAM and leave your graphics card half empty. Keeping the first N expert layers on the GPU instead — same file, different flags:

plan and run compute the cutoff from your free VRAM and emit the flags.

Free speed, part two: if you write code

--spec-type ngram-simple drafts tokens by finding repeated spans in your own context, then verifies them — output is identical, it's one flag, nothing is downloaded.

Draft length is the lever, and it is a kernel decision. Drafts of 4–7 verify in llama.cpp's slow mat-vec path; m≥8 crosses into the fast one. Measured on the same model, same prompt, byte-identical output: 48.2 → 88.5 in one step, up to 132.1 tok/s (5.8×) at m=24.

Copyability is the whole mechanism: code answers repeat their input, prose invents. The 1.03× row is the full expert-offload arm only — on the expert-split placement the quickstart recommends, tuned ngram (--spec-ngram-simple-size-m 384 --spec-ngram-simple-size-n 4) measured 4.7× decode at ~3-bit (21.3 → 98.8 tok/s), shrinking with bit-width (3.4× at Q3_K_M) because the verify round is compute-bound (V-04; preregs #28/#36/#37/#40). Turn it on whenever your output copies its context, on any placement except full expert-offload; on novel generation it drafts nothing and changes nothing.

Measured results

One frame, no cuts: Qwen3-30B-A3B at 20.4 tok/s on a 2016 desktop — GTX 1060 6 GB · 16 GB DDR4 · SATA SSD. Raw logs + GGUF SHA256: EVIDENCE.txt.

Every number above was written down as a prediction, published, and only then measured — including the ones that missed. All predictions and their verdicts → · the four laws behind them →

When quantprobe won't help you

• Your model already fits comfortably in VRAM at 4 bits or more. Community quants are near-lossless there, and — measured — quantizing further buys almost no speed once a model is resident: the same 7B at Q2_K vs Q4_K_M is 36% smaller and 4% slower. Quantize to make a model fit; once it fits, stop. One lever remains inside a fit: on pre-Ampere cards the format sets decode speed — Q4_0 measured +19% end-to-end over Q4_K_M (26.87 vs 22.72 tok/s, preregs #52/#53), and Q2_K was slower than Q4_0 while 32% smaller. Speed-only (Q4_K_M is higher quality per byte), one card measured, unverified on Ampere+ — plan prints it whenever the all-in-VRAM row wins at ≤5.0 bits.
• You want a tight number for a model that fits entirely in VRAM — and you haven't run calibrate. This was the placement the law knew least well, and the ±25% band above does not apply to it. Since v1.20.1 there is a real answer: quantprobe calibrate's all-in-VRAM anchor run plus per-format GPU efficiency (the L-16 format ladder) gives a point prediction for GPU-resident models — ~12% median error across the full ladder, misses erring low, and the anchor's own arm exact by construction (MACHINE_LADDER.md). Uncalibrated, what we can state is one-sided and exception-free: across 8 models and 13 benchmarks, real speed was ≥ 0.90× the printed number every single time, and in 12 of the 13 it was strictly higher — typically 1.1×–1.8×. That is a falsifiable claim with the same logical form as our ±25% band, just asymmetric: one measurement below 0.90× kills it. We have refuted six candidate explanations for the gap, including our own favourites: it is not fixed overhead, not GPU clock state, not bytes-per-token, not monotone in bit-width, not a per-format constant, and not a bytes-weighted mixture of the actual tensor types. Within a single architecture it moves cleanly with the dominant tensor type; across architectures it does not transfer. We would rather publish that than move a constant on thin evidence. This is the single most useful thing you can send us: quantprobe bench --contribute on a GPU-resident model turns your machine into the datapoint that fixes it. One Spark row is already logged against us: Gemma-4-26B reports 0.77× our floor — unexplained, published, next in the queue.
• You need task-level eval scores (MMLU, HellaSwag). quantprobe measures perplexity, KL divergence, and its own 40-task business suite — not academic benchmarks.
• Your architecture isn't in the fragility atlas (four families so far). The probe still works on your model; the published priors just won't apply. Open an issue with your result — those are the most valuable datapoints.
• You want multi-token prediction modeled in the planner. It isn't. Measured, the effect runs from +17% (dense, GPU-resident) to −24% (MoE, experts in RAM) — there's no single multiplier to apply. Full 2×2 →
• You're on a Mac or a 50-series card. Those presets are extrapolated, not measured. quantprobe bench --contribute turns one into a datapoint.
• You need throughput numbers. Everything here is single-stream decode on one machine; expect ±25% across environments.

Commands

quantprobe auto                                # interactive: detects, asks, decides, runs
quantprobe plan  --gguf model.gguf             # predicted tok/s + placement + launch command
quantprobe hw                                  # what the law sees on THIS machine
quantprobe calibrate                           # measure, don't assume: RAM stream, disk, GPU clocks; optional anchor runs
quantprobe run   --gguf model.gguf             # plan the placement, then launch chat
quantprobe audit-ollama                        # what is Ollama's default costing you? measured, contention-guarded
quantprobe bench --gguf model.gguf --contribute # predicted vs measured; opt-in datapoint

quantprobe optimize --tps 20                             # cheapest path to a speed target, Pareto-ranked
quantprobe target   --tps 5 --ladder                     # inverse: target -> smartest model that fits
quantprobe fetch    qwen3-30b ./models                   # robust, resumable download
quantprobe quantize --gguf f16.gguf --out 2bit.gguf      # build a depth-aware quant
quantprobe probe    --gguf f16.gguf --eval wiki.test.raw # measure YOUR model's fragile band
quantprobe dashboard --gguf 2bit.gguf                    # the law live, every reply scored vs prediction

hw/plan/target/optimize need nothing but Python. The weight-touching commands drive stock llama.cpp — point at it with --llama-dir, QUANTPROBE_LLAMA_DIR, or PATH, and preview anything with --dry. 17 machine presets ship in (--machine); multi-GPU and RAID aggregate with comma lists (--vram 24,24).

Windows: 'quantprobe' is not recognized? pip put it in a folder that isn't on your PATH. Use python -m quantprobe ... — identical, always works.

Contributing

quantprobe bench --contribute prints exactly what would be shared plus a pre-filled issue link — you review and submit; nothing is ever sent automatically. Points that land outside the predicted bands are the most valuable ones, and there are open predictions anyone can settle.

Docs

Credits

colibri (744B on 25 GB, pure C) inspired the tier-streaming exploration. The quantization stack builds on llama.cpp and the QTIP/QuIP# incoherence codecs — whose central tool our first law bounds. Independent research by Federico Sciuca, AI-supported, on one desktop.

Two community contributors changed the tool measurably: u/RogerAI--fyi (Reddit) observed that the Law 4 formulation omitted per-token KV reads — measured, confirmed, shipped within a day. u/MoneroApe pointed me at apex-quant and TurboQuant, and testing against mudler's APEX exposed two real gaps in my recipe: unprotected always-active tensors (their kurtosis argument, adopted here) and no importance-matrix calibration at all. MoneroApe then ran the first external replication (RTX 3090 + a 117.6B MoE, register E-06): it exposed five real defects in the shipped tool — the 2× channel-count error, the ubatch cap, a missing pinned-memory warning, a missing --threads, the buried speculation note — all fixed in v1.19 with tests named after the report, and quantprobe calibrate exists because of it.

License

MIT — see LICENSE. © 2026 Federico Sciuca.