nexusquant-kv 0.6.3

NexusQuant

Near-lossless KV cache quantization. Training-free. One line of code.

E8 lattice quantization applied to the KV cache after prefill. No training, no calibration data, no model modifications.

Live demo: huggingface.co/spaces/jmarquex/nexusquant-demo

Headline: on Llama-3.1-8B-Instruct at 4K (n=30, one harness and one needle set), at iso-bpe (2.125 bpe for both once the fp16 scale overhead is counted), NexusQuant K2V2 pb=0 holds 30/30 NIAH while a 2-bit TurboQuant cache drops to 0/30 (NQ-vs-TQ McNemar b=30, c=0; FP16-vs-TQ floor b=29; nq_tq_h2h_llamainst.json). The TurboQuant side is our own reimplementation on our harness, so a faithfulness gap to the reference code cannot be excluded. Near-lossless without eviction: K3V2 pb=0 costs +0.43% PPL on Mistral-Inst-v0.3 (n=161; nq_mistral_subbpe_frontier.json). Validated on 10 architecture families; NIAH recall holds through 32K context (24-25/25 per cell; nq_32k_only.json).

The earlier PyPI/HuggingFace figure of +0.276% PPL on Mistral-7B-v0.1 (n=60, KIVI-iso) is committed in rank116_kivi_iso_bpe.json. The 5.83x compression figure is measured on Mistral-Inst-v0.3 (nq_e8_bpwmatrix.json). PPL deltas above were measured with the 0.5.x quantizer; 0.6.0 corrects the E8 tie-break, so published deltas are conservative for current installs.

Install

pip install "nexusquant-kv[hf]"  # with HuggingFace transformers

compress_kv_cache (quant-only) needs transformers >= 4.46. nexusquant_evict needs transformers >= 5.0 and torch >= 2.4 (uses DynamicLayer hooks).

Quickstart

Prefill all but the last token, compress the cache, then generate.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, DynamicCache
from nexusquant import compress_kv_cache

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1", torch_dtype=torch.float16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
inputs = tokenizer("The KV cache stores key and value tensors for each attention layer.", return_tensors="pt").to(model.device)

cache = DynamicCache()
with torch.no_grad():
    out = model(inputs["input_ids"][:, :-1], use_cache=True, past_key_values=cache)

compressed = compress_kv_cache(out.past_key_values, mode="quant_only", rope_base=getattr(model.config, "rope_theta", 10000.0))
with torch.no_grad():
    print(tokenizer.decode(model.generate(inputs["input_ids"], past_key_values=compressed, max_new_tokens=200, do_sample=False)[0], skip_special_tokens=True))

With eviction (higher compression, needs transformers >= 5.0):

from nexusquant import nexusquant_evict
with nexusquant_evict(model, quality="balanced") as nq:
    output = model.generate(inputs["input_ids"], max_new_tokens=512)

A DynamicCache subclass (E8QuantizedCache) and a kvpress adapter (E8Press) are also available. Full argument reference is in the docstrings.

Results

PPL (quant-only, Mistral-7B-v0.1, wikitext-2, n=60, KIVI-iso)

bpe = (key_bits + value_bits)/2 + 0.125 (the per-head fp16 scale). Source: rank116_kivi_iso_bpe.json.

NIAH recall (quant-only, chat-template, Mistral-Inst-v0.3, A100-80GB)

Sources: nq_32k_only.json, nq_32k_modal_partial.json.

2-bit head-to-head (Llama-3.1-8B-Instruct, 4K, n=30, iso-bpe)

TurboQuant and Block-GTQ are our own reimplementations on our harness; a faithfulness gap to each reference code cannot be excluded.

NQ-vs-TQ McNemar b=30, c=0 (30 discordant pairs, all favor NexusQuant). Sources: nq_tq_h2h_llamainst.json, nq_blockgtq_h2h_4k.json.

Supported architectures

Quality correlates with KV head count. Models with 4+ KV heads are safe at K3V2 (Qwen family needs protect_boundary=2).

Gemma4 (SWA): use compress_layers="global_only". Encoder-decoder, GPT-NeoX/J, and GGUF are not supported.

How it works

1. (optional) Importance-score tokens and evict the lowest, keeping BOS + a recent sliding window
2. Remove RoPE from keys so they share a common subspace
3. Hadamard-rotate to spread outliers across dimensions
4. Quantize each 8-float group to the nearest E8 lattice point (asymmetric: 3-bit keys + 2-bit values)
5. (optional) Boundary-protect first/last N layers at FP16 (mandatory for Qwen-family)

The E8 lattice is the proven densest sphere packing in 8 dimensions and the best known lattice quantizer in 8D (normalized second moment 0.0717 vs 0.0833 for scalar rounding), so the average quantization error per 8-float group is lower than scalar round-to-nearest at the same bit budget. Sources: Viazovska, arXiv:1603.04246 (packing); Agrell and Allen, arXiv:2202.09605 (quantizer, Table I).

Papers

The long-form report is broken into six focused, topic-separated papers.

• Method - paper/nexusquant-method.pdf: the E8 lattice quantizer, the six-stage pipeline, and the quantizer ablations (E8 vs uniform vs Lloyd-Max).
• Architectures - paper/nexusquant-architectures.pdf: does it ruin the language model? Cross-architecture PPL (nine models), prefix-length scaling, mixture-of-experts, multi-turn KV reuse.
• Retrieval - paper/nexusquant-retrieval.pdf: does it remember facts? The 2-bit iso-bpe wedge, FP16 NIAH dead zones, RULER, LongBench, and the eviction cliff.
• Downstream tasks - paper/nexusquant-downstream.pdf: function-call accuracy, HumanEval pass@1, multi-document QA, and GSM8K/MATH reasoning under compression.
• Competitors - paper/nexusquant-competitors.pdf: iso-protocol head-to-heads against KIVI, CommVQ, and AQUA-KV, the non-iso Block-GTQ system comparison, and why the KVTC row was retracted.
• System and negative results - paper/nexusquant-system.pdf: honest byte accounting, rANS entropy coding, latency, memory, the K/V bit grid, and the twelve approaches that did not work.

Limitations

• Eviction hurts factual recall: NIAH degrades past 35% eviction (PPL hides this). Use quant-only when factual accuracy matters.
• Qwen-family models catastrophically fail without protect_boundary=2. Always test your model.
• E8 quantization is CPU-bound; the Triton GPU kernel is written but not yet latency-benchmarked.
• nexusquant_evict does not support batch size > 1 or multi-turn chat; vLLM runtime integration is stubbed (NotImplementedError).

Citation

The work is broken into six focused papers. Cite the one(s) you draw from:

@techreport{nexusquant_method_2026,
  author = {Marques, Jo\~{a}o},
  title  = {{NexusQuant}: Training-Free {E8} Lattice Vector Quantization for {KV} Cache Compression},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-method.pdf},
}
@techreport{nexusquant_arch_2026,
  author = {Marques, Jo\~{a}o},
  title  = {{NexusQuant} Cross-Architecture Perplexity: Does {KV} Cache Compression Preserve Language Modeling?},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-architectures.pdf},
}
@techreport{nexusquant_retrieval_2026,
  author = {Marques, Jo\~{a}o},
  title  = {{NexusQuant} Factual Retrieval: Does {KV} Cache Compression Preserve Needle Recall?},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-retrieval.pdf},
}
@techreport{nexusquant_downstream_2026,
  author = {Marques, Jo\~{a}o},
  title  = {Downstream Task Retention Under {KV} Compression: Function Calls, Code, {Q\&A}, and Reasoning under the {E8} Lattice},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-downstream.pdf},
}
@techreport{nexusquant_competitors_2026,
  author = {Marques, Jo\~{a}o},
  title  = {Calibration-Free {E8} {KV} Cache Quantization versus Competitors: Iso-Protocol Head-to-Head Comparisons},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-competitors.pdf},
}
@techreport{nexusquant_system_2026,
  author = {Marques, Jo\~{a}o},
  title  = {{NexusQuant}: System Cost and Negative Results for Training-Free {E8} {KV} Cache Compression},
  year   = {2026},
  url    = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-system.pdf},
}

License

Apache 2.0. See LICENSE.