tiny-turboquant 0.13.1

Pip-installable compressed KV serving backend and diagnostics for PyTorch

tiny-turboquant

tiny-turboquant is a pip-installable compressed KV-cache serving backend foundation for Hugging Face decoder models.

The package now has two layers:

1. Runtime path: run generation while storing the KV cache compressed between decode steps.
2. Diagnostic path: scan real model KV tensors and validate which K/V policies are safe.

The main direction is no longer “add more benchmarks.” The main direction is:

serve first
validate second
scan third
experimental last

Current status

v0.13.1 is the corrected speed-foundation release for the compressed-KV backend:

Hugging Face model -> prefill -> compress KV -> store compressed between steps -> dequantize before stock HF attention

This is a real compressed-storage decode loop. Incremental KV append and Sparse-V are available as experimental benchmark toggles, but they are default-off because the measured short-context path can be slower. This is not yet a fused compressed-attention accelerator or a production speedup claim.

What this package does now

• Runs a Hugging Face causal LM through a tiny compressed-KV generation engine.
• Stores past_key_values compressed between decode steps.
• Stores KV compressed between decode steps and exposes experimental incremental append with --incremental-kv for A/B benchmarking.
• Supports quality-stable KV policies such as fp16k-q8v and boundary-fp16-fp16k-q8v, plus more aggressive experimental policies.
• Provides an OpenAI-compatible local HTTP server.
• Provides baseline-vs-compressed generation validation with token-match, first-divergence, top-token, and optional logit-delta metrics.
• Keeps the previous real-KV scanner for policy discovery.
• Exposes experimental Sparse-V gating for benchmark experiments; it reports whether SDPA calls were actually intercepted.

What this package does not claim yet

• It does not claim production vLLM speedup.
• It does not claim llama.cpp parity.
• It does not claim fused compressed attention.
• It does not claim TurboQuant+ kernel parity.
• It does not claim that low-bit V is safe for every model.

The current backend stores KV compressed, then dequantizes before calling the stock Hugging Face attention path. Fused dequantization plus attention is a later milestone.

Install

Base package:

pip install tiny-turboquant

For Hugging Face serving:

pip install "tiny-turboquant[hf]"

For server usage:

pip install "tiny-turboquant[server]"

For development:

pip install "tiny-turboquant[dev]"

Quick start

1. List available KV policies

tiny-tq bench --list-policies

Current policies:

Policy	Meaning
fp16	Dense Hugging Face KV cache baseline
fp16k-q8v	Quality-first compressed policy: keep K dense, store V as q8
boundary-fp16-fp16k-q8v	Boundary-protected V-only compression: first/last 2 layers fp16; middle layers fp16 K + q8 V
boundary-fp16-q8k-q8v	First/last 2 layers fp16; middle layers q8 K + q8 V
q8k-q8v	More aggressive q8 K + q8 V storage; can diverge on Qwen-style models
fp16k-int4v	K fp16, V int4 with boundary protection; experimental
q8k-int4v	q8 keys, int4 values, boundary-layer protection; experimental
q8k-turbo4v	serving-facing alias for q8 keys + 4-bit value storage

Start with fp16k-q8v. Recent Qwen validation showed it exact-matched a 64-token prompt while saving about 25% KV memory. Use boundary policies or q8k-q8v only after validation.

2. Run one compressed-KV generation benchmark

tiny-tq bench \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --kv-policy fp16k-q8v \
  --prompt "Explain why KV-cache memory grows during long-context decoding." \
  --max-new-tokens 64

The output includes:

input_tokens
output_tokens
kv_policy
elapsed_seconds
tokens_per_second
cache_report.compressed_storage_bytes
cache_report.dense_fp16_bytes
cache_report.memory_saved_pct

3. Run the deterministic self-test

Before testing compression, verify that the backend loop is deterministic when both sides use dense KV:

tiny-tq validate \
  --self-test \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Answer with one word only. Which city is the capital of France?" \
  --max-new-tokens 5 \
  --quality-threshold 1.0

Expected verdict: exact-match. If this fails, debug the decode loop before testing compression.

4. Compare dense KV against compressed KV

tiny-tq validate \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --baseline fp16 \
  --candidate fp16k-q8v \
  --prompt "Give a short explanation of KV-cache compression." \
  --max-new-tokens 48 \
  --quality-threshold 0.95 \
  --report-json validate_qwen_safe.json \
  --report-md validate_qwen_safe.md

The validation report compares:

baseline generated text
candidate generated text
same prefix token count
first divergence position
token match ratio
prefix match ratio
top-1 token agreement per decode step
optional logit deltas with --collect-logits
baseline cache report
candidate cache report
policy summary
quality verdict

For deeper quality debugging:

tiny-tq validate \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --baseline fp16 \
  --candidate fp16k-q8v \
  --prompt-file prompts/long_prompt.txt \
  --max-new-tokens 64 \
  --collect-logits \
  --report-json validate_long_safe.json \
  --report-md validate_long_safe.md

--collect-logits is useful for debugging but can use much more memory because full vocabulary logits are retained for each compared decode step.

Use --no-per-step when you want compact validation JSON without the full per-step trace. Summary metrics are still computed.

5. Compare multiple policies

Use this after the self-test. It runs several candidate policies against one fp16 baseline and recommends the exact-match compressed policy with the highest memory saving.

tiny-tq compare-policies \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Explain KV-cache compression in simple terms." \
  --max-new-tokens 64 \
  --summary-only \
  --report-json policy_matrix.json \
  --report-md policy_matrix.md

Default policies compared:

fp16
fp16k-q8v
boundary-fp16-fp16k-q8v
boundary-fp16-q8k-q8v
q8k-q8v
q8k-int4v

Recommendation rule:

Choose the exact-match compressed policy with the highest memory saving.
If no compressed policy exact-matches, keep `fp16` as the recommendation and report the best risky compressed candidate separately.

6. Run the local OpenAI-compatible server

tiny-tq serve \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --kv-policy fp16k-q8v \
  --host 127.0.0.1 \
  --port 8000 \
  --max-new-tokens 128

Health check:

curl http://127.0.0.1:8000/health

Chat completion:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain KV-cache compression in two sentences."}
    ],
    "max_tokens": 64
  }'

Completion:

curl http://127.0.0.1:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "KV-cache compression helps because",
    "max_tokens": 64
  }'

Python API

from tiny_turboquant.backends.hf import TinyTurboHFEngine

engine = TinyTurboHFEngine(
    "Qwen/Qwen2.5-0.5B-Instruct",
    kv_policy="fp16k-q8v",
    device="auto",
    dtype="auto",
).load()

result = engine.generate(
    "Explain KV-cache compression in simple terms.",
    max_new_tokens=64,
)

print(result.generated_text)
print(result.cache_report)

Experimental speed toggles

Incremental KV append is default-off in v0.13.1 because the current Python concat/repack path can be slower on short generations. Use it only for A/B benchmarking:

tiny-tq bench \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --kv-policy fp16k-q8v \
  --prompt "Explain KV-cache compression in simple terms." \
  --max-new-tokens 64 \
  --incremental-kv

Sparse-V is also experimental and does not imply speedup:

tiny-tq bench \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --kv-policy fp16k-q8v \
  --prompt "Explain KV-cache compression in simple terms." \
  --max-new-tokens 64 \
  --sparse-v

v0.13.1 speed-foundation release

validate reports whether the compressed backend preserves generation behavior, not just whether it saves memory. compare-policies compares multiple policies and selects the best exact-match compressed policy. If no compressed policy exact-matches, fp16 remains the recommendation and the best risky compressed candidate is reported separately. The default compressed policy is fp16k-q8v.

Key fields:

Field	Meaning
quality_verdict	exact-match, near-stable, prefix-stable-then-diverged, or diverged
comparison.first_divergence_index	First generated token position where baseline and candidate split
comparison.token_match_ratio	Same-token ratio across aligned generated tokens
comparison.prefix_match_ratio	Same-prefix length divided by shorter output length
step_comparison.top1_match_ratio	Agreement between the baseline and candidate top-1 decode choice
logit_comparison	Optional full-logit delta metrics when --collect-logits is used
policy_summary	Memory saving, compression ratio, and candidate/baseline throughput ratio
policy_recommendation	Rule-based status, cause hint, and next policy to try
best_risky_policy	Best compressed policy by token match when no compressed policy exact-matches
unsafe_policies	Compressed policies that changed the generated token path

This is still not a quality guarantee. It is a controlled backend validation harness. Perplexity, NIAH-style tests, and fused-kernel throughput come later.

How the v0.13 backend works

The backend runs a controlled Hugging Face decode loop:

1. Run the prompt through the model with use_cache=True.
2. Compress the full prefill past_key_values into TinyTurboKVCache.
3. Pick the next token.
4. Dequantize the compressed cache back to the model cache format.
5. Run the next token through the model.
6. By default, recompress the current cache after each step because this is currently faster in the measured HF path.
7. With `--incremental-kv`, compress only newly appended KV rows and concatenate them onto the compressed cache when possible.
8. Fall back to dense-dequant/recompress for unsupported layouts.
9. Repeat until max_new_tokens or EOS.

This means the cache is actually stored compressed between decode steps. The incremental append path exists, but it is experimental and default-off because current concat/repack overhead can be slower than full-cache recompression for short generations. The current speed is still not the final goal because dense materialization still happens before stock Hugging Face attention.

Why the default is now quality-first

Real backend validation showed that q8k-q8v can save about 50% KV memory but still diverge early on Qwen-style generation. The quality-stable default therefore protects attention routing first:

first: fp16 vs fp16 self-test
then: fp16k-q8v as current compressed default
more aggressive: boundary-fp16-q8k-q8v
risky until validated: q8k-q8v, q8k-int4v

The key rule is simple: K controls attention routing, so keep K fp16 until a model-specific validation report proves K compression is acceptable.

The package keeps real-model-kv-scan so policy decisions can be measured instead of guessed.

Example scan:

tiny-tq real-model-kv-scan \
  --preset safe \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Long context text..." \
  --max-prompt-tokens 256 \
  --page-size 64 \
  --summary-table \
  --report-json qwen_scan.json \
  --report-md qwen_scan.md

CLI surface

Stable commands:

tiny-tq serve
tiny-tq bench
tiny-tq validate
tiny-tq real-model-kv-scan
tiny-tq kv-estimate
tiny-tq version

Older research commands are still present for compatibility, but they should not define the public product direction. A future cleanup release will move old experiments under an experimental namespace.

Backend roadmap

v0.13.x

• Add fused dequantization plus attention experiments.
• Add long-context sparse-V prototype.
• Add perplexity and prompt-suite validation.

v0.14.x

• Explore vLLM adapter.
• Explore llama.cpp/GGUF bridge or export path.
• Keep Python package install simple while making native acceleration optional.

Development

git clone https://github.com/pradeepboopathy/tiny-turboquant
cd tiny-turboquant
pip install -e ".[dev,hf]"
pytest

Build:

python -m build
python -m twine check dist/*

Upload:

python -m twine upload dist/*

Design rule

tiny-turboquant should be a clean compressed-KV backend, not a pile of experiments.

Experiments are allowed internally. The public path should remain:

serve -> bench -> validate -> scan