llm-dequant 0.1.0

Turn NVFP4-only model releases back into dense safetensors.

Streaming, constant-memory dequantization of compressed-tensors nvfp4-pack-quantized LLM checkpoints — a 300B-parameter MoE converts on a few hundred MB of RAM.

Why this exists

A growing number of community models — abliterations, fine-tunes, supertunes — are published only in NVFP4 compressed-tensors form. That single release format locks out most of the ecosystem:

• llama.cpp's convert_hf_to_gguf.py can't read compressed-tensors checkpoints → no GGUFs
• fine-tuning frameworks expect dense weights → no further training
• weight surgery (MTP grafts, merges, pruning) needs real tensors → no surgery

The reference decompression path in compressed-tensors/transformers materializes the whole model in memory, which is a non-starter at 300B scale.

llm-dequant fixes this with a purpose-built converter:

Born from converting a 295B MoE released as NVFP4-only into a full GGUF quant sweep (SuperHY3-abliterated-GGUF).

Install

pip install llm-dequant            # from PyPI (published releases)

or straight from GitHub (works on any commit, no release needed):

pip install git+https://github.com/vcruz305/llm-dequant.git

Both pull in torch, safetensors, and numpy automatically. For a CPU-only torch (much smaller download):

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install llm-dequant

Quickstart

# 1. what is this checkpoint?
llm-dequant inspect /models/SomeModel-NVFP4

# 2. prove the math round-trips on THIS checkpoint (seconds, no writes)
llm-dequant verify /models/SomeModel-NVFP4 --sample 16

# 3. convert (bf16 default; fp16/fp32 available)
llm-dequant convert /models/SomeModel-NVFP4 /models/SomeModel-BF16

Then use the output like any dense checkpoint:

python llama.cpp/convert_hf_to_gguf.py /models/SomeModel-BF16 \
    --outfile SomeModel-BF16.gguf --outtype bf16

Python API

from llm_dequant.model import convert, verify, inspect_checkpoint
from llm_dequant.nvfp4 import dequantize_nvfp4, unpack_fp4, pack_fp4

convert("/models/in-nvfp4", "/models/out-bf16", dtype="bf16")

The NVFP4 format, in one table

nvfp4-pack-quantized (emitted by llm-compressor) stores, per quantized Linear:

An FP4 code is 1 sign bit + 3 bits indexing the E2M1 magnitude table [0, 0.5, 1, 1.5, 2, 3, 4, 6], and

W[i, j] = e2m1(code[i, j]) · scale[i, j//16] / weight_global_scale

Dequantization is exact. Every recovered value sits precisely on the scaled FP4 lattice (computed in fp32, cast once to the output dtype). Nothing the NVFP4 checkpoint encodes is lost. What the original 4-bit quantization already destroyed is of course not recoverable — the output is dense, not restored. If you re-quantize the result (e.g. to GGUF), disclose the FP4 provenance: tiers at or below ~4 bpw lose essentially nothing, higher tiers only improve the never-quantized tensors.

How correctness is established

Two independent layers, both of which you can run yourself:

1. llm-dequant verify <src> — against your checkpoint. Samples random packed modules, dequantizes, renormalizes by the effective scale, re-packs with nearest-E2M1 rounding, and requires byte equality with the original weight_packed. Any error in nibble order, scale layout, group mapping, or global-scale handling breaks byte equality. (This proves self-consistency against the real bytes; spec conformance comes from layer 2.)

2. The test suite — against the reference implementation. Hand-computed vectors, exhaustive FP4 code tables, property-based round-trips, end-to-end synthetic checkpoints (sharded + single-file, with/without global scale), corruption and path-traversal rejection, and a bit-exactness test against upstream compressed-tensors (runs in CI).

pip install -e .[dev] && pytest

Scope

Supported: format: nvfp4-pack-quantized, symmetric, group_size: 16 — the only layout llm-compressor emits for NVFP4 today. Fused-MoE expert tensors ([experts, m, n/2]) work unchanged.

Rejected loudly, never converted silently: other compressed-tensors formats, asymmetric quantization (weight_zero_point), NaN/Inf scales, shard names that escape the checkpoint directory, colliding output tensor names.

Dropped by design: activation-quantization metadata (input_global_scale) — the output is dense; runtime activation quantization no longer applies.

License

Apache-2.0. The FP4/E2M1 constant table follows the format specification established by the vLLM / compressed-tensors projects (Apache-2.0).