tiny-turboquant 0.1.0

Low-bit vector and KV-cache compression research toolkit for PyTorch

Tiny TurboQuant

Tiny TurboQuant is a lightweight PyTorch research toolkit for low-bit vector compression and KV-cache compression experiments.

It includes:

• real uint8 bit-packing for low-bit indices
• MSE-style scalar quantization
• product / inner-product-oriented quantization experiments
• outlier-split quantization
• a Hugging Face-compatible KV-cache prototype
• demos for ANN search, embedding compression, and KV-cache memory measurement

Important limitation

This package demonstrates packed memory compression. It is not a production compressed-attention engine. The KV-cache wrapper still dequantizes tensors before attention. Real latency gains require fused CUDA/Triton kernels or integration with an inference engine such as vLLM or TensorRT-LLM.

Install from local wheel

pip install tiny_turboquant-0.1.0-py3-none-any.whl

Basic usage

import torch
from tiny_turboquant import TurboQuantMSE, TurboQuantKVCache

x = torch.randn(128, 64)
x = x / x.norm(dim=-1, keepdim=True)

q = TurboQuantMSE.build(d=64, bits=4)
idx = q.quant(x)
x_hat = q.dequant(idx)

cache = TurboQuantKVCache(bits=4)

Run tests

python -m pytest

Run demos from source checkout

python -m demos.demo1_distortion_vs_theory
python -m demos.demo2_ann_vs_pq
python -m demos.demo3_real_embeddings
python -m demos.demo4_kv_cache

Scope

Use this package for:

• compressed vector-search experiments
• RAG embedding compression experiments
• KV-cache memory/quality tradeoff experiments
• educational or research benchmarking

Do not claim it provides production-speed LLM inference. It reduces packed storage; speed requires optimized compressed-attention kernels.