tiny-turboquant 0.3.0

Low-bit vector and KV-cache compression research toolkit for PyTorch

Tiny TurboQuant

Tiny TurboQuant is a lightweight PyTorch research toolkit for low-bit vector compression and KV-cache compression experiments.

Version 0.3.0 adds quality-hardening and integration-oriented research features:

• optional per-head calibration in HybridTurboQuantKVCache
• separate Key/Value recent windows for stronger Key preservation
• safer outlier clamping for small head dimensions
• paged/streaming attention utility that avoids one giant dense cache reconstruction in research tests
• CompressedVectorIndex API with optional full-precision reranking
• serving-engine integration scaffolding for vLLM/TensorRT-LLM style experiments

Important limitation

This package demonstrates packed memory compression and memory-quality benchmarking. It is not a production compressed-attention engine. Hugging Face generation still receives dense K/V tensors. The paged attention utility dequantizes page-by-page and avoids one full dense cache tensor, but it is not a fused CUDA/Triton kernel. Real latency gains require fused kernels or serving-engine integration.

Do not use this package to claim training acceleration, fine-tuning memory reduction, production legal/medical QA readiness, drop-in vLLM replacement, exact nearest-neighbor search, or faster LLM inference.

Install

pip install tiny-turboquant

Hybrid KV-cache usage

from tiny_turboquant import HybridTurboQuantKVCache

cache = HybridTurboQuantKVCache(
    key_bits=6,
    value_bits=4,
    key_outlier_bits=8,
    value_outlier_bits=8,
    n_key_outliers=32,
    n_value_outliers=16,
    key_recent_window=128,
    value_recent_window=64,
    per_layer_calibration=True,
    per_head_calibration=True,
)

Compressed vector index usage

import torch
from tiny_turboquant import CompressedVectorIndex

emb = torch.randn(10_000, 384)
index = CompressedVectorIndex(bits=4, store_original_for_rerank=True).add(emb)
results = index.search(emb[0], top_k=5, rerank_top_k=100)

print(index.compression_ratio())
print(results[0])

Paged attention utility

import torch
from tiny_turboquant import HybridTurboQuantKVCache

cache = HybridTurboQuantKVCache(key_bits=6, value_bits=4, recent_window=16)
k = torch.randn(1, 4, 64, 32)
v = torch.randn(1, 4, 64, 32)
cache.update(k, v, layer_idx=0)

q = torch.randn(1, 4, 1, 32)
out = cache.paged_attention(q, layer_idx=0)

Serving integration scaffolding

from tiny_turboquant import PagedKVCacheSpec, VLLMExperimentAdapter

spec = PagedKVCacheSpec(key_bits=6, value_bits=4, page_size=128)
plan = VLLMExperimentAdapter(spec).integration_plan()
print(plan)

Scope

Use this package for:

• compressed vector-search experiments
• RAG embedding compression experiments
• KV-cache memory/quality tradeoff experiments
• educational or research benchmarking
• testing hybrid recent-token retention strategies
• studying separate K/V compression settings
• testing per-layer/per-head calibration

Current focus:

• memory compression
• quality measurement
• cache-format experiments
• research scaffolding for serving integration

Future direction:

• actual Triton/CUDA fused dequant + attention kernels
• stronger compressed-domain attention estimators
• engine-specific paged-cache integration experiments