tiny-turboquant 0.12.0.2

Pip-installable compressed KV serving backend and diagnostics for PyTorch

tiny-turboquant

tiny-turboquant is a pip-installable compressed KV-cache serving backend foundation for Hugging Face decoder models.

The package now has two layers:

1. Runtime path: run generation while storing the KV cache compressed between decode steps.
2. Diagnostic path: scan real model KV tensors and validate which K/V policies are safe.

The main direction is no longer “add more benchmarks.” The main direction is:

serve first
validate second
scan third
experimental last

Current status

v0.12.0.1 introduces the first compressed-KV backend foundation:

Hugging Face model -> generate token-by-token -> store KV compressed between steps -> dequantize before stock HF attention

This is a real compressed-storage decode loop. It is not yet a fused compressed-attention accelerator.

What this package does now

• Runs a Hugging Face causal LM through a tiny compressed-KV generation engine.
• Stores past_key_values compressed between decode steps.
• Supports conservative KV policies such as q8k-q8v and q8k-int4v.
• Provides an OpenAI-compatible local HTTP server.
• Provides baseline-vs-compressed generation validation.
• Keeps the previous real-KV scanner for policy discovery.

What this package does not claim yet

• It does not claim production vLLM speedup.
• It does not claim llama.cpp parity.
• It does not claim fused compressed attention.
• It does not claim TurboQuant+ kernel parity.
• It does not claim that low-bit V is safe for every model.

The current backend stores KV compressed, then dequantizes before calling the stock Hugging Face attention path. Fused dequantization plus attention is a later milestone.

Install

Base package:

pip install tiny-turboquant

For Hugging Face serving:

pip install "tiny-turboquant[hf]"

For server usage:

pip install "tiny-turboquant[server]"

For development:

pip install "tiny-turboquant[dev]"

Quick start

1. List available KV policies

tiny-tq bench --list-policies

Current policies:

Policy	Meaning
fp16	Dense Hugging Face KV cache baseline
q8k-q8v	Conservative compressed K and V storage
q8k-int4v	q8 keys, int4 values, boundary-layer protection
q8k-turbo4v	serving-facing alias for q8 keys + 4-bit value storage

Start with q8k-q8v. Move to q8k-int4v only after validation.

2. Run one compressed-KV generation benchmark

tiny-tq bench \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --kv-policy q8k-q8v \
  --prompt "Explain why KV-cache memory grows during long-context decoding." \
  --max-new-tokens 64

The output includes:

input_tokens
output_tokens
kv_policy
elapsed_seconds
tokens_per_second
cache_report.compressed_storage_bytes
cache_report.dense_fp16_bytes
cache_report.memory_saved_pct

3. Compare dense KV against compressed KV

tiny-tq validate \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --baseline fp16 \
  --candidate q8k-q8v \
  --prompt "Give a short explanation of KV-cache compression." \
  --max-new-tokens 48 \
  --report-json validate_qwen_q8.json

The validation report compares:

baseline generated text
candidate generated text
same prefix token count
exact token match
baseline cache report
candidate cache report

4. Run the local OpenAI-compatible server

tiny-tq serve \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --kv-policy q8k-q8v \
  --host 127.0.0.1 \
  --port 8000 \
  --max-new-tokens 128

Health check:

curl http://127.0.0.1:8000/health

Chat completion:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain KV-cache compression in two sentences."}
    ],
    "max_tokens": 64
  }'

Completion:

curl http://127.0.0.1:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "KV-cache compression helps because",
    "max_tokens": 64
  }'

Python API

from tiny_turboquant.backends.hf import TinyTurboHFEngine

engine = TinyTurboHFEngine(
    "Qwen/Qwen2.5-0.5B-Instruct",
    kv_policy="q8k-q8v",
    device="auto",
    dtype="auto",
).load()

result = engine.generate(
    "Explain KV-cache compression in simple terms.",
    max_new_tokens=64,
)

print(result.generated_text)
print(result.cache_report)

How the v0.12 backend works

The backend runs a controlled Hugging Face decode loop:

1. Run the prompt through the model with use_cache=True.
2. Compress the returned past_key_values into TinyTurboKVCache.
3. Pick the next token.
4. Dequantize the compressed cache back to a dense legacy past_key_values tuple.
5. Run the next token through the model.
6. Compress the updated cache again.
7. Repeat until max_new_tokens or EOS.

This means the cache is actually stored compressed between decode steps. The current speed is not the final goal because dense materialization still happens before attention.

Why the default is conservative

Real model-query validation on Qwen showed that random-query diagnostics were too optimistic. Lower-bit V could look good under random queries and fail under real model queries. The safer initial backend policy is therefore asymmetric and conservative:

start: q8k-q8v
then validate: q8k-int4v
later: fused/sparse/turbo policies

The package keeps real-model-kv-scan so policy decisions can be measured instead of guessed.

Example scan:

tiny-tq real-model-kv-scan \
  --preset safe \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt "Long context text..." \
  --max-prompt-tokens 256 \
  --page-size 64 \
  --summary-table \
  --report-json qwen_scan.json \
  --report-md qwen_scan.md

CLI surface

Stable commands:

tiny-tq serve
tiny-tq bench
tiny-tq validate
tiny-tq real-model-kv-scan
tiny-tq kv-estimate
tiny-tq version

Older research commands are still present for compatibility, but they should not define the public product direction. A future cleanup release will move old experiments under an experimental namespace.

Backend roadmap

v0.12.x

• Stabilize the HF compressed-KV backend.
• Improve generation validation.
• Add per-layer fallback explanations.
• Add q8k-int4v and q8k-turbo4v validation reports.

v0.13.x

• Add fused dequantization plus attention experiments.
• Add long-context sparse-V prototype.
• Add perplexity and prompt-suite validation.

v0.14.x

• Explore vLLM adapter.
• Explore llama.cpp/GGUF bridge or export path.
• Keep Python package install simple while making native acceleration optional.

Development

git clone https://github.com/pradeepboopathy/tiny-turboquant
cd tiny-turboquant
pip install -e ".[dev,hf]"
pytest

Build:

python -m build
python -m twine check dist/*

Upload:

python -m twine upload dist/*

Design rule

tiny-turboquant should be a clean compressed-KV backend, not a pile of experiments.

Experiments are allowed internally. The public path should remain:

serve -> bench -> validate -> scan