tiny-turboquant 0.13.5

Pip-installable compressed KV serving backend and diagnostics for PyTorch

tiny-turboquant

tiny-turboquant is a pip-installable compressed KV-cache serving backend foundation for Hugging Face decoder models.

The package now has two layers:

1. Runtime path: run generation with selectable cache modes: lazy-shadow for fp16-parity decode plus end-of-generation compression, dense-shadow for per-step compressed-at-rest reporting, or compressed mode for step-by-step compressed-cache materialization.
2. Diagnostic path: scan real model KV tensors and validate which K/V policies are safe.

The main direction is no longer “add more benchmarks.” The main direction is:

serve first
validate second
scan third
experimental last

Current status

v0.13.5 promotes cache_mode="lazy-shadow" to the default TinyTurboHFEngine cache mode. New callers now get fp16-parity active decode out of the box, with the compressed KV store built once at end-of-generation. The previous default (dense-shadow) is unchanged and still selectable via cache_mode="dense-shadow". Generation reports now also expose cache_report["compressed_at_rest_available"] ("end-of-generation" for lazy-shadow, "per-step" for dense-shadow and compressed).

v0.13.4 is a measured decode-speed patch for cache_mode="compressed". Vectorizing the int-packed KV unpack path (replacing a per-bit Python loop of scatter_add_ / shift / fill kernels with a single fused op, plus an 8-bit fast path that skips bit-shuffling entirely) gives a 4.6× decode speedup in per-step compressed mode on Kaggle T4 (Qwen2.5-1.5B-Instruct, fp16k-q8v, 1K prompt, 32 decode tokens): 1.52 → 6.93 tok/s. GPU profile confirms the old hot path (aten::scatter_add_ at 55% of CUDA time, 14k calls per 32 decode steps) is now gone; the new top is aten::mm + scaled_dot_product_attention as it should be.

See the v0.13.4 compressed-mode speedup section for the full before/after profile and the honest tradeoff.

v0.13.4 also adds a new cache_mode="lazy-shadow". It holds only the dense past_key_values reference during decode and compresses once at end-of-generation, eliminating the step-0 compression cost and the per-step shadow refresh of dense-shadow. On Kaggle T4 (Qwen2.5-1.5B-Instruct, fp16k-q8v, 1K prompt, 32 decode tokens), lazy-shadow runs at 12.25 tok/s vs the fp16 baseline 12.22 tok/s — at-parity active decode with compressed-at-rest reporting available after generation finishes. v0.13.4 also silences the Transformers ≥4.45 torch_dtype → dtype deprecation warning by selecting the kwarg via inspect.signature.

v0.13.3 is a follow-on validation patch that adds measured Kaggle T4 numbers for the three "last-N" policies shipped in v0.13.2 (fp16k-int4v-last8-fp16, q8k-int4v-last8-q8v, fp16k-int4v-boundary-last). All 24 cells in the sweep produced bit-exact greedy generation; see the last-N section below.

v0.13.2 is the dense-shadow speed-foundation release for the compressed-KV backend:

Hugging Face model -> prefill -> keep fp16 active shadow for decode -> rebuild/report compressed KV store

The default lazy-shadow mode runs Hugging Face decode on the stock fp16 cache path and compresses once at end-of-generation. The legacy dense-shadow mode also keeps decode close to the stock fp16 path while producing per-step compressed KV storage reports. Both are useful for validating compressed-at-rest KV policies; neither is active compressed attention, and neither is a production speedup claim.

For lower active KV residency, use --cache-mode compressed. That mode materializes from compressed storage each step and is expected to be much slower with stock Hugging Face attention.

What this package does now

• Runs a Hugging Face causal LM through a tiny compressed-KV generation engine.
• Supports three cache modes: lazy-shadow for fp16-parity decode with end-of-generation compressed reporting, dense-shadow for fast active decode with per-step compressed-at-rest reporting, and compressed for storage-only step-by-step materialization.
• Stores/report past_key_values in compressed form and separates compressed-at-rest savings from active decode memory.
• Supports quality-stable KV policies such as fp16k-q8v and boundary-fp16-fp16k-q8v, plus more aggressive experimental policies.
• Provides an OpenAI-compatible local HTTP server.
• Provides baseline-vs-compressed generation validation with token-match, first-divergence, top-token, and optional logit-delta metrics.
• Keeps the previous real-KV scanner for policy discovery.
• Exposes experimental Sparse-V gating for benchmark experiments; it reports whether SDPA calls were actually intercepted.

What this package does not claim yet

• It does not claim production vLLM speedup.
• It does not claim llama.cpp parity.
• It does not claim fused compressed attention.
• It does not claim TurboQuant+ kernel parity.
• It does not claim that low-bit V is safe for every model.

The current backend stores KV compressed, then dequantizes before calling the stock Hugging Face attention path. Fused dequantization plus attention is a later milestone.

Install

Base package:

pip install tiny-turboquant

For Hugging Face serving:

pip install "tiny-turboquant[hf]"

For server usage:

pip install "tiny-turboquant[server]"

For development:

pip install "tiny-turboquant[dev]"

Quick start

1. List available KV policies

tiny-tq bench --list-policies

Current policies:

Policy	Meaning
fp16	Dense Hugging Face KV cache baseline
fp16k-q8v	Quality-first compressed policy: keep K dense, store V as q8
boundary-fp16-fp16k-q8v	Boundary-protected V-only compression: first/last 2 layers fp16; middle layers fp16 K + q8 V
boundary-fp16-q8k-q8v	First/last 2 layers fp16; middle layers q8 K + q8 V
q8k-q8v	More aggressive q8 K + q8 V storage; can diverge on Qwen-style models
fp16k-int4v	K fp16, V int4 with boundary protection; experimental
q8k-int4v	q8 keys, int4 values, boundary-layer protection; experimental
fp16k-int4v-last8-fp16	int4 V except last 8 layers kept fp16 K/V; experimental last-N protection
q8k-int4v-last8-q8v	q8 K + int4 V except last 8 layers q8 K/V; experimental last-N protection
fp16k-int4v-boundary-last	boundary + last-N protected int4 V policy; experimental
q8k-turbo4v	serving-facing alias for q8 keys + 4-bit value storage

Start with fp16k-q8v. Recent Qwen validation showed it exact-matched a 64-token prompt while saving about 25% KV memory. Use boundary policies or q8k-q8v only after validation.

2. Run one compressed-KV generation benchmark

tiny-tq bench \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --kv-policy fp16k-q8v \
  --prompt "Explain why KV-cache memory grows during long-context decoding." \
  --max-new-tokens 64

The output includes:

input_tokens
output_tokens
kv_policy
elapsed_seconds
tokens_per_second
cache_report.compressed_storage_bytes
cache_report.dense_fp16_bytes
cache_report.memory_saved_pct
cache_report.compressed_at_rest_saved_pct
cache_report.active_decode_memory_saved_pct
cache_report.cache_mode

3. Run the deterministic self-test

Before testing compression, verify that the backend loop is deterministic when both sides use dense KV:

tiny-tq validate \
  --self-test \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Answer with one word only. Which city is the capital of France?" \
  --max-new-tokens 5 \
  --quality-threshold 1.0

Expected verdict: exact-match. If this fails, debug the decode loop before testing compression.

4. Compare dense KV against compressed KV

tiny-tq validate \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --baseline fp16 \
  --candidate fp16k-q8v \
  --prompt "Give a short explanation of KV-cache compression." \
  --max-new-tokens 48 \
  --quality-threshold 0.95 \
  --report-json validate_qwen_safe.json \
  --report-md validate_qwen_safe.md

The validation report compares:

baseline generated text
candidate generated text
same prefix token count
first divergence position
token match ratio
prefix match ratio
top-1 token agreement per decode step
optional logit deltas with --collect-logits
baseline cache report
candidate cache report
policy summary
quality verdict

For deeper quality debugging:

tiny-tq validate \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --baseline fp16 \
  --candidate fp16k-q8v \
  --prompt-file prompts/long_prompt.txt \
  --max-new-tokens 64 \
  --collect-logits \
  --report-json validate_long_safe.json \
  --report-md validate_long_safe.md

--collect-logits is useful for debugging but can use much more memory because full vocabulary logits are retained for each compared decode step.

Use --no-per-step when you want compact validation JSON without the full per-step trace. Summary metrics are still computed.

5. Compare multiple policies

Use this after the self-test. It runs several candidate policies against one fp16 baseline and recommends the exact-match compressed policy with the highest memory saving.

tiny-tq compare-policies \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Explain KV-cache compression in simple terms." \
  --max-new-tokens 64 \
  --summary-only \
  --report-json policy_matrix.json \
  --report-md policy_matrix.md

Default policies compared:

fp16
fp16k-q8v
boundary-fp16-fp16k-q8v
boundary-fp16-q8k-q8v
q8k-q8v
q8k-int4v
fp16k-int4v-last8-fp16
q8k-int4v-last8-q8v
fp16k-int4v-boundary-last

Recommendation rule:

Choose the exact-match compressed policy with the highest memory saving.
If no compressed policy exact-matches, keep `fp16` as the recommendation and report the best risky compressed candidate separately.

6. Run the local OpenAI-compatible server

tiny-tq serve \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --kv-policy fp16k-q8v \
  --host 127.0.0.1 \
  --port 8000 \
  --max-new-tokens 128

Health check:

curl http://127.0.0.1:8000/health

Chat completion:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain KV-cache compression in two sentences."}
    ],
    "max_tokens": 64
  }'

Completion:

curl http://127.0.0.1:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "KV-cache compression helps because",
    "max_tokens": 64
  }'

Python API

from tiny_turboquant.backends.hf import TinyTurboHFEngine

engine = TinyTurboHFEngine(
    "Qwen/Qwen2.5-0.5B-Instruct",
    kv_policy="fp16k-q8v",
    device="auto",
    dtype="auto",
).load()

result = engine.generate(
    "Explain KV-cache compression in simple terms.",
    max_new_tokens=64,
)

print(result.generated_text)
print(result.cache_report)

v0.13.2 speed-foundation release

validate reports whether the compressed backend preserves generation behavior, not just whether it saves memory. compare-policies compares multiple policies and selects the best exact-match compressed policy. If no compressed policy exact-matches, fp16 remains the recommendation and the best risky compressed candidate is reported separately. The default compressed policy is fp16k-q8v.

Key fields:

Field	Meaning
quality_verdict	exact-match, near-stable, prefix-stable-then-diverged, or diverged
comparison.first_divergence_index	First generated token position where baseline and candidate split
comparison.token_match_ratio	Same-token ratio across aligned generated tokens
comparison.prefix_match_ratio	Same-prefix length divided by shorter output length
step_comparison.top1_match_ratio	Agreement between the baseline and candidate top-1 decode choice
logit_comparison	Optional full-logit delta metrics when --collect-logits is used
policy_summary	Memory saving, compression ratio, and candidate/baseline throughput ratio
policy_recommendation	Rule-based status, cause hint, and next policy to try
best_risky_policy	Best compressed policy by token match when no compressed policy exact-matches
unsafe_policies	Compressed policies that changed the generated token path

This is still not a quality guarantee. It is a controlled backend validation harness. Perplexity, NIAH-style tests, and fused-kernel throughput come later.

Measured throughput and quality (v0.13.2)

Sweep across Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct on Kaggle T4, prompts of ~1K / 2K / 4K tokens, 64 decode tokens, greedy. fp16 baseline = 1.00.

Throughput ratio vs fp16 (active decode):

model	seq	fp16k-q8v	boundary-fp16-fp16k-q8v	boundary-fp16-q8k-q8v	q8k-q8v	q8k-int4v
Qwen2.5-1.5B	1K	1.07	1.16	1.08	1.06	1.10
Qwen2.5-1.5B	2K	0.95	1.01	0.90	0.88	0.91
Qwen2.5-1.5B	4K	0.93	0.95	0.85	0.82	0.88
Qwen2.5-3B	1K	1.12	1.15	1.10	1.07	1.10
Qwen2.5-3B	2K	0.96	0.98	0.90	0.88	0.92
Qwen2.5-3B	4K	0.92	0.93	0.85	0.84	0.88

Storage saving and quality:

policy	KV bytes saved	quality (greedy, all cells)
fp16k-q8v	22.7%	exact-token match
boundary-fp16-fp16k-q8v	19.4–20.1%	exact-token match
boundary-fp16-q8k-q8v	38.8–40.3%	exact-token match
q8k-q8v	45.3%	exact-token match
q8k-int4v	56.0–56.4%	exact-token match

Every cell in the matrix produced bit-exact greedy generation against the fp16 baseline. The minimum throughput ratio anywhere is 0.82; the recommended fp16k-q8v default holds ≥0.92 at every (model, seq-len) tested. The 1K rows showing ratios above 1.0 are most likely warmup variance on short runs and should not be read as a "faster than fp16" claim.

What the savings mean

The compressed KV store is reported every step, but during active decode the engine holds a fp16 "dense shadow" of the cache (dense_shadow_during_decode: true in cache_report) so the model runs against its own fp16 past at full speed. The compressed store is rebuilt once at end of decode for accurate memory_saved_pct. Memory savings therefore apply at rest — between requests, on KV swap-out, in cross-tenant pooling, and for cache serialization — not to peak active-decode resident memory. For workloads that actually pay the KV-storage bill (multi-tenant serving, long-context KV swap), the saving is real; for single-stream peak-memory budgeting, plan around fp16.

To force the slow per-step dequant path (drops shadow, lets you measure the storage-only mode):

tiny-tq bench --kv-policy fp16k-q8v --cache-mode compressed ...

v0.13.4 compressed-mode decode speedup

cache_mode="compressed" runs the per-step dequant path that dequantizes the int-packed KV cache before each decode call. It is the storage-minimum mode for memory-constrained serving (KV swap, multi-tenant pools). Through v0.13.3 it was significantly slower than dense-shadow because the bit-packing path ran a per-bit Python loop on the GPU — packing or unpacking bits-wide values issued bits separate scatter_add_ / index_select / shift kernel launches per call.

v0.13.4 replaces that loop with:

• An 8-bit fast path: q8 values are stored one-per-byte, so pack is idx.to(uint8) and unpack is a reshape. Zero scatter ops, zero shifts. This is the path fp16k-q8v (the default compressed policy) hits.
• A vectorized general path for 1-7 bit widths that computes all bits per-value bit contributions in one fused tensor op and issues one scatter_add_ / one index_select instead of bits of them.

Measured on Kaggle T4

Model: Qwen2.5-1.5B-Instruct, policy fp16k-q8v, 1K WikiText-103 prompt, 32 decode tokens, greedy, single GPU.

	fp16 (dense-shadow)	compressed v0.13.3	compressed v0.13.4
tok/s	13.91	1.52	6.93
Self CUDA total	986 ms	9.56 s	1.22 s
Self CPU total	2.31 s	21.12 s	4.63 s
scatter_add_ CUDA	n/a	5.23 s (55%)	gone
aten::fill_ calls	12,493	332,057	21,257
__rshift__ + __lshift__ calls	0	~104,832	0
top CUDA op (compressed)	n/a	scatter_add_ (dequant)	aten::mm (linears)

That is a 4.6× decode-speed improvement in compressed mode (1.52 → 6.93 tok/s) and a 7.9× reduction in GPU compute time, without any quality change — pack/unpack is bit-exact.

Honest position vs dense-shadow

After v0.13.4 the compressed-mode profile is healthy: ~71% of GPU time is now actual model compute (aten::mm 40%, scaled_dot_product_attention 31%), with the remaining ~30% being dequant materialization, scale apply, int8→fp16 cast, and GQA broadcast. Per-op GPU cost in compressed vs fp16 is essentially identical (mm: 489 vs 484 ms; SDPA: 372 vs 352 ms). The remaining gap to fp16 throughput (6.93 vs 13.91 tok/s) is Python/CPU launch overhead from per-layer dequant orchestration, not GPU work — GPU utilization is 26% in compressed vs 43% in fp16.

This means:

• For active-decode throughput, lazy-shadow (the default since v0.13.5) is the recommendation — it runs the stock fp16 cache path during decode and compresses once at end-of-generation. dense-shadow remains available when per-step compressed-at-rest reporting is required.
• For peak-resident KV memory (KV swap, multi-tenant cache pooling, large-batch serving), cache_mode="compressed" is now a defensible tradeoff at ~0.5× fp16 throughput on T4, vs the ~0.11× it was through v0.13.3.

Closing the remaining gap is a v0.14.x research item (CUDA graphs for the decode loop, batched cross-layer dequant) and does not require writing a fused attention kernel — SDPA is not the bottleneck.

Reproduce

from torch.profiler import profile, ProfilerActivity, record_function
from tiny_turboquant.backends.hf import TinyTurboHFEngine
import torch

eng = TinyTurboHFEngine(
    "Qwen/Qwen2.5-1.5B-Instruct",
    kv_policy="fp16k-q8v",
    cache_mode="compressed",   # swap to "dense-shadow" for the fp16 baseline row
    device="auto", dtype="auto",
).load()

PROMPT = open("prompt_1k.txt").read()
_ = eng.generate(PROMPT, max_new_tokens=4)            # warmup
torch.cuda.synchronize()

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
    with record_function("decode_total"):
        r = eng.generate(PROMPT, max_new_tokens=32)
torch.cuda.synchronize()

print(f"tok/s={r.tokens_per_second:.2f}")
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=15))

Last-N policy validation (v0.13.3)

v0.13.2 shipped three "last-N" policies that preserve the most recent 8 tokens at higher precision (motivated by the observation that recent KVs dominate near-term attention). v0.13.3 measures them on the same matrix (Qwen2.5-{1.5B,3B} on Kaggle T4, ~1K/2K/4K prompts from WikiText-103, 64 decode tokens, greedy). fp16 baseline = 1.00.

Throughput ratio vs fp16:

model	seq	fp16k-int4v-last8-fp16	q8k-int4v-last8-q8v	fp16k-int4v-boundary-last
Qwen2.5-1.5B	1K	1.08	1.02	1.13
Qwen2.5-1.5B	2K	1.07	0.97	1.08
Qwen2.5-1.5B	4K	1.01	0.88	1.00
Qwen2.5-3B	1K	1.16	1.11	1.26
Qwen2.5-3B	2K	1.03	0.93	1.03
Qwen2.5-3B	4K	0.98	0.89	0.97

Storage saving and quality:

policy	KV bytes saved	quality (greedy, all cells)
fp16k-int4v-last8-fp16	25.1–27.3%	exact-token match
q8k-int4v-last8-q8v	54.2–55.0%	exact-token match
fp16k-int4v-boundary-last	29.1–30.4%	exact-token match

All 24 cells produced bit-exact greedy output. fp16k-int4v-boundary-last is the new sweet spot when you want both extra recent-token protection (last-8 fp16) and the boundary-fp16 prefix anchor on top of int4 values: it stays within 0.97×–1.26× of fp16 throughput while saving ~30% of KV bytes. The q8k variant trades ~5–12% throughput for the headline ~55% saving and is the right choice when KV bytes — not active speed — is the limit. Numbers may differ slightly from the v0.13.3 table above because the prompts are drawn from a different corpus (WikiText-103 here vs the v0.13.3 internal prompt set); the ratios and exact-match behavior are what generalize.

How the v0.13 backend works

The backend has three explicit cache modes.

lazy-shadow mode — fp16-parity (default)

The fastest mode. The engine holds only the dense past_key_values reference during decode and runs token-by-token generation on the stock Hugging Face cache path with no per-step compression or shadow refresh. The compressed KV store is built once at end-of-generation:

1. Run the prompt through the model with use_cache=True.
2. Wrap the dense past_key_values reference (no compression yet).
3. Run token-by-token generation using the stock Hugging Face cache path.
4. Compress the final past_key_values into TinyTurboKVCache at end-of-generation.
5. Report compressed-at-rest savings.

Use this when you want fp16-parity active decode and only need compressed-at-rest numbers after generation finishes. On Kaggle T4 with Qwen2.5-1.5B-Instruct + fp16k-q8v, lazy-shadow matched the fp16 baseline at 12.25 vs 12.22 tok/s.

dense-shadow mode — per-step compressed reporting

This is the speed-friendly validation mode used for the v0.13.3 throughput table.

1. Run the prompt through the model with use_cache=True.
2. Keep the model's fp16 past_key_values as the active decode shadow.
3. Run token-by-token generation using the dense Hugging Face cache path.
4. Rebuild/report the compressed KV store at the end of decode.
5. Report both compressed-at-rest savings and active decode memory fields.

This mode validates compressed KV storage policies without paying per-step dequantization cost. It is fast, but active decode still holds a fp16 shadow cache. Its memory savings apply to compressed-at-rest use cases such as cache swap-out, cache serialization, pooling, and post-decode storage.

compressed mode

This is the lower-active-storage experimental path.

1. Run the prompt through the model with use_cache=True.
2. Compress the full prefill past_key_values into TinyTurboKVCache.
3. Before each next-token forward, materialize the model cache from compressed storage.
4. Compress/append the new KV rows when possible.
5. Repeat until max_new_tokens or EOS.

This mode proves the compressed cache can drive generation, but it is slow with stock Hugging Face attention because it still materializes dense tensors before attention. Fused compressed attention is the later v0.14 direction.

Use:

tiny-tq bench --kv-policy fp16k-q8v --cache-mode compressed ...

Why the default is now quality-first

Real backend validation showed that q8k-q8v can save about 50% KV memory but still diverge early on Qwen-style generation. The quality-stable default therefore protects attention routing first:

first: fp16 vs fp16 self-test
then: fp16k-q8v as current compressed default
more aggressive: boundary-fp16-q8k-q8v
risky until validated: q8k-q8v, q8k-int4v

The key rule is simple: K controls attention routing, so keep K fp16 until a model-specific validation report proves K compression is acceptable.

The package keeps real-model-kv-scan so policy decisions can be measured instead of guessed.

Example scan:

tiny-tq real-model-kv-scan \
  --preset safe \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Long context text..." \
  --max-prompt-tokens 256 \
  --page-size 64 \
  --summary-table \
  --report-json qwen_scan.json \
  --report-md qwen_scan.md

CLI surface

Stable commands:

tiny-tq serve
tiny-tq bench
tiny-tq validate
tiny-tq real-model-kv-scan
tiny-tq kv-estimate
tiny-tq version

Older research commands are still present for compatibility, but they should not define the public product direction. A future cleanup release will move old experiments under an experimental namespace.

Backend roadmap

v0.13.x

• Add fused dequantization plus attention experiments.
• Add long-context sparse-V prototype.
• Add perplexity and prompt-suite validation.

v0.14.x

• Explore vLLM adapter.
• Explore llama.cpp/GGUF bridge or export path.
• Keep Python package install simple while making native acceleration optional.

Design rule

tiny-turboquant should be a clean compressed-KV backend, not a pile of experiments.

Experiments are allowed internally. The public path should remain:

serve -> bench -> validate -> scan