PrismaQuant

Mixed-precision LLM quantization that chooses the right format for every weight matrix, selected on real end-to-end KL — shipped as artifacts that stock inference engines serve on an unforked runtime.

PrismaQuant's allocator is AURA (Production-Faithful KL–Fisher Allocation): a per-Linear cost model built from KL-Fisher probes of the full model multiplied against the production-rendered weight error — the exact bytes that ship — solved as a multi-choice knapsack, and gated by real KL measured on a held-out split before anything is published. Three output containers:

• compressed-tensors — vanilla vLLM serves it natively (vllm serve $WORK_DIR/exported), no custom kernels. NVFP4 / FP8 / BF16 per Linear, CUTLASS kernels on Blackwell.
• GGUF — llama.cpp and vLLM (via the GGUF plugin) serve the same file, again with no PrismaQuant kernels. Full k-quant + IQ menu, per-tensor mixed, imatrix-weighted. This is how a 295B MoE fits on one 128 GB box.
• Codebook (NVFP4-CB / FP8-CB) — served by gridbook, our out-of-tree vLLM quantization plugin, using native CUDA/CUTLASS kernels only. Still an unforked vLLM: install the exact commit in prismaquant/gridbook_runtime/gridbook_runtime_pin.json, no core patches; a missing required native operation fails closed. See below.

Headlines

All quality numbers below are served-artifact measurements (exact vLLM KL-vs-BF16 on held-out text, or deterministic benchmark runs on the served endpoint) — never local screens.

Qwen3.6-27B PrismaAURA 5.5 bpp — tool-use fidelity above full precision.

Served KL-vs-BF16 0.0342 — a −40.9% reduction over the prior AURA build at the same bpp, from adding the FP8 middle rung to the menu plus render/export fidelity fixes. rdtand/Qwen3.6-27B-PrismaAURA-5.5bit-vllm

Ornith-1.0-35B-A3B PrismaAURA 4.75 bpp — 70 GB → 23 GB, KL 0.0143.

Served confident KL-vs-BF16 0.0143 (top-1 agreement 98.6%), with a grafted MTP head for speculative decoding (91.3% acceptance at position 0). rdtand/Ornith-1.0-35B-PrismaAURA-4.75bit-vllm-MTP

Qwen3.6-35B-A3B 4.75 bpp — wins 8 of 9 zero-shot metrics vs uniform NVFP4.

Against RedHatAI's uniform NVFP4 quantization (342 hand-picked BF16 ignores), the measured allocation ships 2 GB smaller with ~90 fewer Linears in BF16 and lands ~4× closer to BF16 on mean zero-shot delta (−0.56 pp vs −2.21 pp). Uniform precision collapses the ~5% of genuinely sensitive Linears; measurement finds them. rdtand/Qwen3.6-35B-A3B-PrismaQuant-4.75bit-vllm

Tencent Hy3 295B-A21B → 103.7 GB GGUF — serves on a single DGX Spark.

A 295B/21B-active MoE quantized from the BF16 source to 2.80 bpp by measured allocation over the GGUF k-quant + IQ menu (streaming probe — the model never fits in memory; byte-budget selection targets the box, not a curve heuristic). No quality claims — models at this scale can't be KL-validated against their BF16 teacher on the target hardware; validation is load + coherent-generation smokes and bit-exact packing. rdtand/Hy3-295B-A21B-PrismaQuant-2.8bit-gguf-vllm, plus a 5.3-bit compressed-tensors variant with MTP for two Sparks.

The artifact family is at ~400k downloads on Hugging Face (rdtand).

Quick start

compressed-tensors lane (vLLM):

export MODEL_PATH=/path/to/model
export WORK_DIR=./dq-runs/mymodel
export FORMATS=NVFP4,FP8_DYNAMIC,BF16   # the production menu — FP8 belongs in every recipe
export TARGET_BITS=4.75
# COST_MODE=aura is the DEFAULT since 2026-07-30 (KL-Fisher x production-rendered dW);
# set COST_MODE=production-render-score to reproduce pre-flip artifacts.
export SELECTION_MODE=validated-surrogate  # real-KL frontier selection (default: surrogate)

./prismaquant/run-pipeline.sh
vllm serve $WORK_DIR/exported --quantization compressed-tensors

GGUF lane (llama.cpp + vLLM):

export EXPORT_CONTAINER=gguf TARGET_PROFILE=gguf COST_MODE=local
export PRODUCTION_CACHE=0 PRODUCTION_RECACHE=0
export FORMATS=IQ2_XS,IQ3_XXS,IQ4_XS,Q4_K,Q5_K,Q6_K,Q8_0
export TARGET_BITS=2.9

./prismaquant/run-pipeline.sh

Every stage is also runnable on its own; the allocator takes the same menu as a flag, so a re-solve against existing probe/cost artifacts needs no re-measurement:

python -m prismaquant.allocator --formats NVFP4,FP8_DYNAMIC,BF16 \
  --probe      $WORK_DIR/artifacts/probe.pkl \
  --costs      $WORK_DIR/artifacts/cost.pkl \
  --layer-config $WORK_DIR/artifacts/layer_config.json \
  --pareto-csv $WORK_DIR/artifacts/pareto.csv \
  --target-bits 4.75

The pipeline runs probe → cost → allocator → export → serve-smoke end-to-end, with fail-fast gates on any configuration that would break the measurement contract (it will tell you exactly why and what to set). Models too large for memory (200B+ MoE) run the streaming layer-by-layer path — peak memory is bounded by ~1 layer + a tunable cache.

How AURA works

Mixed-precision quantization splits into two orthogonal questions:

• Local (well-studied): given a fixed format, how do you round this one Linear best? GPTQ, scale search, activation ordering — the per-tensor toolkit runs under whatever format is chosen.
• Global (PrismaQuant's contribution): how many bits should each Linear get, and in which hardware format? A per-Linear allocation of a total bit budget across the format menu.

AURA answers the global question with three commitments:

1. A KL-faithful per-Linear cost. The classical additive cost is ½ · H_trace · MSE_W — a Fisher-diagonal trace times weight round-trip error. AURA replaces both halves: the sensitivity comes from KL-Fisher probes of the full model (the adjoint of the end-to-end KL objective, not a layer-local proxy), and the error term is measured on the production-rendered weights — GPTQ, joint scale optimization, activation ordering, the exact render that ships — not a raw round-trip. On served A/Bs at matched bpp this cost beats the strong h_trace × output_mse baseline by −38% KL on a 4B model and −17.9% on 27B (replicated across two calibration corpora).

2. Production-faithfulness as an invariant. The cost the allocator measures, the KL the validator measures, and the bytes the exporter ships all come from one weight cache — no rendering confound anywhere in the loop. On the GGUF lane the same contract holds through the imatrix: the calibration weighting used to measure a format's cost is byte-identical to the weighting used to pack it (enforced bit-exact against gguf-py, the llama.cpp reference decoder).

3. Surrogates generate, real KL selects. The additive surrogate is a candidate generator, never the ship decision. The allocator renders a Pareto sweep of candidates, each is measured with real end-to-end KL on a held-out split (disjoint from everything the cost stage saw), and the shipping point is picked on the measured (bpp, KL) frontier. Outside the surrogate's trust region, bpp order ≠ KL order: on the 27B, the surrogate's own knee picks 5.86 bpp / KL 0.056 while validated selection picks 5.31 bpp / KL 0.015 — smaller and 3.7× better, which is what end-to-end measurement buys over trusting the model of the cost.

MoE hybrid. Smooth per-Linear costs are structurally blind to router flips: quantizing a routed expert can change which experts fire, and no local error term sees that. AURA therefore allocates non-expert weights from the smooth cost and packed routed experts from measured empirical unit-KL (each expert-tier choice scored end-to-end), merged into one knapsack.

What we measured and dropped. The cross-layer-interaction literature (CLADO's pairwise IQP, HAWQ-V3's second-order ILP, cooperative-game formulations) models the coupling between Linears. We built the full L2 perturbed-activation cascade and ran the head-to-head on the served metric: cross-layer modeling bought −1.5%; AURA's better per-Linear cost bought −38.5% on the same A/B. Per-Linear cost fidelity was the lever; interaction modeling was not. That experiment — and the rest of the rejected-methods catalog (pairwise QUBO, Lagrangian λ-bisection as a selector, top-K Hessian covering, polish DPs that regress under the real-KL gate) — is documented in the paper. Negative results are recorded with their durable lesson; we don't re-litigate them silently.

Full derivations, the additivity/cancellation analysis, and the served evidence: paper/main.pdf.

Pipeline

incremental_probe ──────► probe.pkl        (KL-Fisher sensitivity per Linear; streaming for 200B+)
        │
cost stage ─────────────► cost.pkl         (COST_MODE=aura: KL-Fisher × production-rendered dW;
        │                                   MoE hybrid merges measured expert unit-KL)
allocator ──────────────► layer_config.json + Pareto candidates
        │                                  (multi-choice knapsack; fused-sibling & packed-MoE
        │                                   format coherence via union-find promotion)
validate_assignments_kl ► held-out real-KL per candidate
select_validated_frontier► the shipping point, picked on measured (bpp, KL)
        │
export ─────────────────► exported/        (compressed-tensors  OR  GGUF via export_gguf)
        │
validate_native_export ─► engine load + greedy-decode smoke (eager and graph mode)
validate_quantized_model► PPL / p99 per-prompt NLL / MTP-acceptance ship gate

Every hot path is GPU-bound by design; the pipeline refuses to run on CPU.

Formats

compressed-tensors container (vLLM-native):

GGUF container (llama.cpp + vLLM GGUF plugin):

GGUF quantizers are PrismaQuant's own GPU implementations (imatrix-weighted grid/least-squares search, exhaustive codeword search for IQ), validated bit-exact against gguf-py. GPTQ-under-frozen-scales is available for the k-quants as a research lever. The allocator is serving-profile-aware in both containers: it never picks a format the target engine can't serve at that tensor's shape.

Codebook lane (gridbook)

Below NVFP4's 4.5 bpw floor the vLLM-native menu runs out of rungs, and the GGUF IQ formats that fill the gap carry a large prefill tax on this hardware. The NVFP4-CB / FP8-CB product-codebook formats open measured 2–6 bpw rungs with a served kernel behind them: the resident weight is a packed k-bit index stream plus a tiny shared codebook. Native CUDA decodes small-M work directly; larger-M quality paths use only bounded per-layer transient expansion consumed by CUTLASS, so no dense [N,K] weight remains resident.

Serving is the separately released gridbook package: an out-of-tree vLLM quantization plugin (registry key gridbook, legacy read alias prismaquant) with zero vLLM-core patches. PrismaQuant 0.8.0 pins Gridbook 0.8.0 at exact commit 9011a19228ddb96b8a49e11a20ac75c99c83998e. That runtime JIT-builds its native CUDA support kernels and CUTLASS GEMM/grouped-GEMM kernels at model load, attests every required operation and shape before the first forward, and fails closed if the native path is unavailable. It has no Gridbook Triton dependency, dispatch arm, or serving fallback, and it bypasses vLLM's fallback-capable convenience wrappers. Export is EXPORT_CONTAINER=nvfp4_cb (prismaquant/export_nvfp4_cb.py). PrismaQuant contains no copy of the runtime: its only supported integration source is the immutable commit in prismaquant/gridbook_runtime/gridbook_runtime_pin.json, and CI compares producer profiles/rungs/layouts to that install's packaged runtime_contract.json. Gridbook may publish the same source as a release, but PrismaQuant serving and validation never replace the commit pin with a moving or merely version-matched package request. DeepSeek-V4/DSV4 remains absent from that consumer contract and is not qualified by this release.

Public artifact: rdtand/Hy3-295B-A21B-gridbook-2.9bit-vllm — 295B/21B-active at 2.9 bpw on one DGX Spark, prefill 89 tok/s against the same model's GGUF IQ artifact at 42 tok/s. No directional quality claim at this scale, same as the GGUF lane. Lane record: docs/lanes/nvfp4-cb/.

Architectures

First-class profiles in-tree:

• Qwen3 / 3.5 / 3.6 — original Qwen3 dense + 30B-A3B routed MoE under Gridbook producer id qwen3; 3.5/3.6 packed-3D MoE + MTP heads and Gated-DeltaNet hybrids
• Tencent Hy3 (hy_v3) — 295B/21B-active, 192-expert MoE + MTP sidecar
• DeepSeek-V4-Flash (vendored transformer + profile) — ~285B total by checkpoint arithmetic, 281,263,734,784 probe-measured quantizable parameters (166.9 GB on disk), per-expert-Linear MoE. The 671B headline belongs to the DeepSeek-V3 family and does not describe Flash; size claims here are per-checkpoint from the probe inventory, never the family headline.
• MiniMax M2 / M2.7 — nested per-expert MoE, native-FP8 source
• Gemma4 — multi-layer-type rope, KV sharing, visual/text profiles
• LFM2.5 — per-expert MoE + short-conv layers
• poolside Laguna (laguna) — 117B/256-expert sparse MoE, separate DFlash drafter as the MTP-analog

A new architecture is a model_profiles/ registration (structure spec JSON + a small profile class); most of the fused-group and naming plumbing is auto-derived from the vLLM model class, so ports land in ~30–200 LoC.

Published artifacts

Method discipline

The rules this project holds itself to, because low-bit quantization results are unusually easy to overstate:

• Promote on the serving metric, not the screen. A win counts only when it holds on exact full-vocab vLLM KL-vs-BF16 and direct perplexity on the served artifact at matched bpp. We have watched local-PPL "wins" invert on the served A/B more than once; those methods are archived with the lesson, not quietly dropped.
• KL screens, it doesn't ship alone. Lower mean KL can hide a heavier tail; candidates also clear direct PPL, p99 per-prompt NLL, and tool-use benchmarks before publication.
• Held-out means held-out. Selection KL uses text the cost stage never saw.
• No hand-tuned bans. If the allocator picks something that breaks, the cost model is wrong — fix the measurement, don't constrain the optimizer. The only constants allowed are ones derived from a dtype's numerical precision.
• No quality claims we can't back. The 295B artifact ships with an explicit "no quality claims" section because its teacher can't be measured on the target hardware. Provenance (git commit, calibration hash, assignment hash) is baked into every artifact's metadata.

Documentation

• docs/ARCHITECTURE.md — the system map: stages, contracts, containers, where each decision is actually made.
• docs/README.md — index of the rest: normative design rules and runtime flags (docs/design/), per-container lane records (docs/lanes/), dated results and audits, and the archive.

Citation

@software{prismaquant2026,
  title  = {PrismaQuant: Mixed-Precision LLM Quantization Selected on Real End-to-End KL},
  author = {Tand, Robert},
  year   = {2026},
  url    = {https://github.com/RobTand/prismaquant},
}

The AURA paper (AURA: Production-Faithful KL–Fisher Allocation) is at paper/main.pdf (source: paper/main.tex). Contact: robert.tand@icloud.com.

Acknowledgements

PrismaQuant is assembled on top of a decade of quantization research. Key influences: CLADO (Deng et al. 2023) for the cross-layer decision-unit framing (whose interaction-modeling half our head-to-head then retired); HAWQ-V1–V3 (Dong et al. 2019–2021), CoopQ, and AMQ on mixed-precision allocation; GPTQ (Frantar et al. 2022) and its Babai-nearest-plane interpretation (Chen et al. 2026) for rounding; the llama.cpp/ggml project for the GGUF formats and reference implementations; Kneedle (Satopää et al. 2011); and Cover & Thomas ch. 13 on rate-distortion bit allocation. Full bibliography in paper/main.tex.