voicequant 0.0.1

TurboQuant KV cache compression for voice AI inference — 5x more concurrent sessions on the same GPU

VoiceQuant

5x More Concurrent Voice Agents on the Same GPU

VoiceQuant applies TurboQuant KV cache compression to cut KV memory 5x, enabling 50 concurrent voice agent sessions on a single T4 GPU instead of 10.

The Problem

Voice AI inference is expensive. Each concurrent caller needs their own KV cache. On a T4 (16GB), after loading a 7B model (~4GB), you have ~12GB for KV caches. At FP16, that's only ~8 concurrent sessions at 4K context. Add a 1500-token system prompt and you're looking at even fewer.

The Solution

VoiceQuant uses TurboQuant (PolarQuant rotation + Lloyd-Max quantization + QJL residual correction) to compress KV caches from 16-bit to 4-bit with 0.99+ cosine similarity. Same GPU, ~40 concurrent sessions.

Quick Start

pip install voicequant

# Start serving (requires GPU + vLLM)
voicequant serve --model Qwen/Qwen2.5-7B-Instruct-AWQ

# Validate compression quality
voicequant verify --model Qwen/Qwen2.5-7B-Instruct-AWQ --bits 4

# Run voice AI benchmarks
voicequant bench --all --report benchmark_report.md

Benchmark Results

Metric	FP16	TQ4 (4-bit)	TQ3 (3-bit)	Improvement
Concurrent sessions (T4, 4K ctx)	~8	~40	~55	5x
KV cache per session (4K ctx)	~150 MB	~30 MB	~22 MB	5-7x smaller
TTFB at 8K context	baseline	~same	~same	neutral
Key cosine similarity	1.000	0.993+	0.985+	-
Value cosine similarity	1.000	0.990+	0.975+	-
Tool calling accuracy	100%	~100%	~99%	-

Deploy to Modal (One Command)

# Generate deployment files
voicequant deploy modal --model Qwen/Qwen2.5-7B-Instruct-AWQ --gpu T4

# Deploy
modal deploy deploy/modal_deploy.py

Your endpoint is now live at https://your-workspace--voicequant.modal.run/v1.

Deploy to RunPod

voicequant deploy runpod --model Qwen/Qwen2.5-7B-Instruct-AWQ --gpu T4

Deploy with Docker

voicequant deploy docker --model Qwen/Qwen2.5-7B-Instruct-AWQ
docker compose up --build

Use with LiveKit Agents

VoiceQuant exposes an OpenAI-compatible API, so any agent framework works as a drop-in:

from livekit.agents import AgentSession, Agent, JobContext, WorkerOptions, cli
from livekit.plugins import deepgram, openai, cartesia, silero

VOICEQUANT_URL = "https://your-workspace--voicequant.modal.run/v1"

async def entrypoint(ctx: JobContext):
    await ctx.connect()

    session = AgentSession(
        vad=silero.VAD.load(),
        stt=deepgram.STT(model="nova-3"),
        llm=openai.LLM(
            model="Qwen/Qwen2.5-7B-Instruct-AWQ",
            base_url=VOICEQUANT_URL,
            api_key="voicequant",
        ),
        tts=cartesia.TTS(model="sonic-3"),
    )

    await session.start(
        agent=Agent(instructions="You are a helpful voice assistant."),
        room=ctx.room,
    )

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

Use with Inference Gateway

Add VoiceQuant as a provider in your gateway config:

# gateway.yaml
models:
  llm:
    voicequant/qwen2.5-7b-tq4:
      provider: openai_compatible
      base_url: https://your-voicequant.modal.run/v1
      api_key: voicequant

Supported Models

Model	Size	AWQ Variant	Weights RAM	Voice Quality	Recommended GPU
Qwen2.5-3B-Instruct	3B	AWQ 4-bit	~2GB	Good for simple tasks	T4 (16GB)
Qwen2.5-7B-Instruct	7B	AWQ 4-bit	~4GB	Excellent for voice	T4/A10G
Llama-3.1-8B-Instruct	8B	AWQ 4-bit	~5GB	Great all-around	T4/A10G
Mistral-7B-Instruct-v0.3	7B	AWQ 4-bit	~4GB	Good instruction following	T4/A10G
Qwen2.5-14B-Instruct	14B	AWQ 4-bit	~8GB	Best quality in class	A10G/L4

Concurrent Session Estimates

GPU	Memory	Model Weights	Available for KV	FP16 Sessions	TQ4 Sessions
T4	16 GB	~4 GB	~12 GB	~8	~40
A10G	24 GB	~4 GB	~20 GB	~13	~65
L4	24 GB	~4 GB	~20 GB	~13	~65
A100	80 GB	~4 GB	~76 GB	~50	~250
H100	80 GB	~4 GB	~76 GB	~50	~250

How TurboQuant Works

1. PolarQuant Rotation: A fixed random orthogonal matrix rotates KV cache coordinates so they become approximately Gaussian distributed.
2. Lloyd-Max Quantization: Optimal scalar quantization for Gaussian data. Provably minimizes MSE for the given bit budget.
3. QJL Residual Correction (keys only): Random projection of the quantization residual preserves inner product expectations, correcting bias in attention scores.

Result: 3-4 bits per element with 0.99+ cosine similarity.

• Keys: 2-bit MSE quantization + 1-bit QJL bias correction (3 bits total)
• Values: 3-bit MSE quantization
• Both compressed in a single fused kernel per attention head

Voice-Specific Optimizations

• Residual window (default: 256 tokens): Recent tokens stay in FP16 for maximum quality. Older tokens (system prompt, early conversation) get compressed aggressively.
• Low max_tokens (default: 150): Voice responses should be 1-3 sentences. A 500-token response takes 10+ seconds to speak.
• Continuous batching: vLLM's continuous batching handles many concurrent short sessions efficiently.
• Streaming by default: TTFB matters more than throughput for voice AI.

CLI Reference

# Start server
voicequant serve --model MODEL --tq-bits 4 --port 8000

# Run benchmarks
voicequant bench --all --report output.md
voicequant bench --scenario concurrent --max-sessions 50
voicequant bench --scenario multi_turn

# Validate quality
voicequant verify --model MODEL --bits 4 --threshold 0.99

# Deploy
voicequant deploy modal --model MODEL --gpu T4
voicequant deploy runpod --model MODEL --gpu T4
voicequant deploy docker --model MODEL

API Endpoints

Endpoint	Method	Description
/v1/chat/completions	POST	OpenAI-compatible chat completions (streaming + non-streaming)
/v1/models	GET	List available models
/v1/health	GET	Health check with GPU memory status
/v1/capacity	GET	Estimated concurrent session capacity
/v1/kv-stats	GET	KV cache memory usage and compression ratio
/metrics	GET	Prometheus-format metrics

Development

pip install -e ".[all]"
pytest tests/ -v

Acknowledgments

• TurboQuant — Google Research (ICLR 2026): PolarQuant rotation + Lloyd-Max + QJL residual correction
• DevTechJr/turboquant-gpu — cuTile CUDA kernels + PyTorch fallback
• Alberto-Codes/turboquant-vllm — vLLM plugin integration
• 0xSero/turboquant — Standalone implementation
• mitkox/vllm-turboquant — vLLM fork with Triton backend

License

Apache 2.0