TurboQuant Pro

Consumer-aware compression for embedding indexes and LLM KV caches. TurboQuant Pro compresses each vector by the metric its downstream consumer actually uses — retrieval recall for indexes, attention/generation quality for KV caches — not reconstruction cosine alone, which is repeatedly shown here to be blind, or even anti-correlated, with quality.

pip install turboquant-pro
tqp replay embedding_glove_recall --small   # reproduce the headline retrieval claim — CI-gated, runs in seconds

• Embedding retrieval: on public data, preregistered and scored by rule across six arms (GloVe-100, deep-image-96, NYTimes-256, DBpedia ada-002 1M, DBpedia text-embedding-3-large 1M, Wikipedia-1024 10M; RaBitQ in four forms, OPQ and PQ at matched stored bytes), tq-pro wins at the low-byte end on low-dimensional data, ties RaBitQ and OPQ once the rerank saturates, and loses only where its own kernel wrapped or at one 132-byte Wikipedia point — benchmarks/RESULTS_rabitq_public.md, ledger status reproducible. The private-sample headline — 32× compression at recall@10 ≈ 0.999, a statistical tie with OPQ and above RaBitQ on a 199k LaBSE sample, 4–20× cheaper to build than OPQ (RaBitQ itself builds faster) — is a single reported run, corroborated at 1M on Gutenberg. The CI-gated number is ~9.6× at recall@10 ≈ 0.999 on GloVe-100 (tqp replay embedding_glove_recall).
• KV caches: architecture-aware key quantization avoids a failure that is invisible to reconstruction metrics — PolarQuant keys read 0.995 cosine yet blow perplexity to ≈10⁴; per-channel keys keep it near fp16.
• At scale & in production: compressed-domain search, persisted / larger-than-RAM sharded + memory-mapped indexes, distribution-free rank certificates, one-command replay, and drift monitoring.

Every headline number — with its reproduction status, dataset, one-click notebook, and hardware — is a row in CLAIMS.md. The acceptance signal everywhere is rank fidelity / a certificate / the consumer's metric — never reconstruction cosine.

The latest published pre-release is 2.0.0a3 (scan kernel v3, bit widths that follow the spectrum, residual-coded IVF, the quantization control plane, the public RaBitQ verdicts, the empirical false-clear rate, the reconciled claims ledger, the stratified instruments); 2.0.0a2 brought production semantics for the vLLM KV connector, the Postgres track and the self-metering anatomy instruments. The last stable line is 1.9.x (larger-than-RAM search + index format v3), and the tqp CLI and certification platform shipped in 1.8.0. APIs under connectors/ may still move before 2.0.0. Full notes: CHANGELOG.md.

Installation

pip install turboquant-pro          # core (numpy only) + the `tqp` CLI
pip install turboquant-pro[torch]   # + operator tracer (`tqp trace`)
pip install turboquant-pro[fast]    # + AVX2 ADC kernel (pybind11)
pip install turboquant-pro[gpu]     # + CuPy CUDA 12.x
pip install turboquant-pro[analysis]  # + readscope + the tqp-readscope bridge
                                    #   (blind read-operator measurement)
pip install turboquant-pro[all]     # everything (pgvector, FAISS, NATS, …)

30-second embedding compression

The central, best-validated contribution — compress a corpus and search the codes directly:

from turboquant_pro import PCAMatryoshka, ADCIndex

pca = PCAMatryoshka(input_dim=768, output_dim=256).fit(train_vectors)
pipeline = pca.with_quantizer(bits=3)                 # PCA rotate/truncate + 3-bit TurboQuant
index = ADCIndex(pipeline).add(corpus)                # compressed-domain index (100 B/vec: 256×3 bits + a 4-byte norm)

ids, scores = index.search(queries, k=10)                          # single-pass, fast
ids, scores = index.search(queries, k=10, rerank=5, originals=corpus)   # exact rerank → ~0.9997

PCAMatryoshka.suggest_output_dim(corpus, target_variance=0.95) picks the truncation dim from the data's spectrum. Two more ways to spend the same bytes:

pipeline = pca.with_spectrum_quantizer(budget_bytes=100)   # widths 1–4 bits per dim, allocated over the PCA spectrum, same 100 B

from turboquant_pro import IVFIndex
ivf = IVFIndex.create(corpus, output_dim=256, bits=3, nlist=1024)   # k-means cells over the same codes, residual-coded
ids, scores = ivf.search(queries, k=10, nprobe=32)                   # scans the probed cells only

The spectrum allocation beat uniform widths at four of six byte levels on 1024-d Wikipedia and lost at every level on flat-spectrum GloVe-100 (benchmarks/RESULTS_spectrum_bits.md, preregistered: S1 MIXED, S2 REFUTED); residual-coded IVF raised single-pass recall@10 over the flat index at the same bytes by 0.02–0.08 on GloVe-100, 0.07–0.19 on deep-image-96 and 0.01–0.02 on NYTimes-256, and by under 0.01 on the 1024- and 1536-d arms, where thousands of centroids shorten a residual little. See the user guide.

Or compress an LLM KV cache

Architecture-aware by quantizer, not just bit-width — per-channel keys + PolarQuant values:

from turboquant_pro import TurboQuantKVCache

cache = TurboQuantKVCache.robust(head_dim=128, n_heads=32, hot_window=512)  # asym-NF4 keys + 2% outliers, 4-bit K/V
# or auto-configure from a model name:
from turboquant_pro import AutoConfig
cache = AutoConfig.from_pretrained("llama-3-8b", target="balanced").build_cache()   # K4/V3

robust() is one codebook that stays near-fp16 across every architecture tested (including high-GQA models where symmetric NF4 silently collapses). See the KV keys finding.

Choose your workflow

Why consumer-aware compression?

One governing principle ties the whole toolkit together:

Compress a tensor by the metric its consumer uses. Accept or reject on that metric — recall, perplexity, a rank certificate, an expert-set flip rate — never reconstruction cosine on its own.

The sharpest illustration is KV-cache keys. PolarQuant normalizes each key and quantizes its direction, discarding the per-channel scale that softmax(Q·Kᵀ) depends on. On Qwen2.5 that reads a reassuring 0.995 key cosine while perplexity explodes to ≈10⁴; per-channel key quantization at the same width keeps it near fp16 (≈15). A reconstruction-only benchmark cannot see this. Full write-up: docs/KV_KEYS_FINDING.md.

That boundary is now instrumented, so the principle ships as tooling rather than advice:

• rank_certificate — turns a measured distortion κ + the corpus's distance-ratio concentration μ̂ into a distribution-free rank floor (Kendall τ ≥ 1−2μ̂); a vacuous floor is the per-corpus "exact reranking required" signal. Emit with tqp certify, re-check with tqp verify (a third party re-hashes the inputs and reproduces the math).
• a2_probe — selects the quantizer family against the declared consumer (cosine / L2 / attention logits) at calibration time; it reproduces the keys catastrophe as a unit test.
• operator_trace / operator_sensitivity — infer each tensor's consumer (softmax score / residual / MoE gate / SSM decay) and apply the discipline that operator needs, validated on real Mixtral, OLMoE, and Mamba models.
• read_operators — the consumer's read operator P_C itself, as a pluggable provider, so the consumer-relative distortion tr(P_C·Σ_δ) is a number you gate on rather than a principle you appeal to. identity (which is reconstruction error, made explicit), declared, and attention_analytic (a closed form, since a head's read subspace is spanned by its queries) ship in tree; tqp-readscope adds blind recovery for consumers with no closed form. Why a registry and not a function: two defensible references for one attention head differ by about 0.3 in overlap, so a number is not interpretable unless it names the reference it was computed against — tqp certify --reference PROVIDER records exactly which one, hashed.
• read_allocation — spends a bit budget against that operator by reverse water-filling, which is the same optimization as power allocation across frequency bins with a task's sensitivity in place of a signal's power. allocation_report gives the gain over uniform at a fixed budget, and warns when the operator's spectrum is so concentrated that the split rests on one or two directions — which is correct if the operator is, and is also exactly what a sub-dimensionally recovered operator looks like regardless of the consumer. If you are allocating for a long context, allocate against the operator estimated over the whole sequence rather than a calibration prefix — that was better on every cell C-11c measured.

Backed by the companion theory papers: the-angular-observer (the rank-certificate and (A2) transfer theory) and geometric-observation — the evidence repository home of Paper III (Observation Theory: consumer-relative rate–distortion and the omission floor) and Paper IV (the consumer-relative flip). TurboQuant Pro is Paper II of that series, the compression-as-observation work.

The measurement side ships separately as readscope (pip install readscope, or pip install turboquant-pro[analysis] for the whole stack) — it measures which directions of a vector a computation actually reads, recovering P_C from a consumer's outputs alone. Its core is numpy by default and backend-generic by dispatch: hand it CuPy arrays and the linear algebra runs on-GPU with seed-identical probe directions, and the tqp-readscope bridge ingests torch tensors zero-copy via DLPack (CPU zero-copy always; CUDA zero-copy into CuPy when present, else exactly one warned host copy). The packages stay deliberately independent — neither imports the other; the bridge is the only code that knows both. readscope's specification is worth reading before trusting a measured operator: recovery against the direction budget is a cliff at k = d, rank-independent — originally a measured law, since proved as a theorem (confined transcripts cannot identify hidden components below full dimension; see readscope's PRINCIPLES.md, P3) — so a sub-dimensional probe returns the dominant direction and little else, and no faster backend moves that cliff, only the FLOPs around it.

The strategic bet

As models and vector databases scale, the binding constraint shifts from storing the vector to preserving what its consumer reads with it. Reconstruction fidelity — the objective essentially every quantizer optimizes — is increasingly the wrong one: it can show a reassuring 0.995 cosine while the downstream task collapses. TurboQuant Pro is the production embodiment of the alternative: measure the consumer's read operator, spend bits against it, and ship a certificate that the ranking survives — turning a theory program (Paper I's transfer/rank theory, Paper IV's consumer-relative flip) into instruments you run in CI. The bet is that certified, consumer-aware compression becomes table stakes as ratios climb and silent quality regressions get more expensive to miss. That is the axis this project competes on — not one more point on the compression-vs-reconstruction curve, but the certificate that the compression preserved the thing that mattered.

How it works

A per-vector flow — extract L2 norm → unit-normalize → random-orthogonal rotate → Lloyd-Max scalar-quantize → bit-pack — compresses embeddings and KV-cache values near-losslessly (the TurboQuant algorithm, Zandieh et al., ICLR 2026). KV-cache keys take the per-channel path instead (above).

flowchart LR
    A["Raw vector<br/>(float32)"] --> B["PCA-Matryoshka<br/>rotate + truncate"]
    B --> C["Random orthogonal<br/>rotation"]
    C --> D["TurboQuant<br/>Lloyd-Max SQ"]
    D --> E["Bit-pack<br/>8×3-bit = 3 B"]
    E --> F["Compressed code"]
    A -. "L2 norm (kept alongside)" .-> F
    classDef out fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px;
    class F out;

The scan (kernel v3, 2.0.0a3). ADCIndex keeps its codes in the AVX2 kernel's own blocked layout — 32-row blocks, two 4-bit codes per byte — packed once at add() and never repacked at search; each add() batch is a chunk, and a search is one kernel call over the chunks a query probes with a per-(query, chunk) constant, so an IVF cell is the same object with a centroid. The lookup table is per dimension and dimensions group into weighted segments, so widths that differ across the spectrum scan in one pass. Measured on Atlas (Wikipedia-1024, 1M rows, PCA-512 at 3 bits, 8 threads): 264 B/row from 520, build 28 s from 34, 16.0 ms per query from 18.9, top-10 identical on every query. The 2.0.0a2 kernel wrapped its sums past 257 dimensions; that fix (3d96506) is in this release and is why the full-dimension text-embedding-3-large cells of the public comparison lost.

Benchmark snapshot

At 32× compression (96 B/vec vs 3072 B fp32; the tq-pro production row is 100 B), recall@10 on a real 199k LaBSE sample — all methods reranked identically (5× oversample + exact rerank). This is a single reported run on a private file (benchmarks/RESULTS_labse_199k.md, ledger row embedding_labse_32x_headline); the public-data version of the same table is one Run all of the canonical notebook, and the only CI-gated retrieval number is the GloVe row in CLAIMS.md.

Bootstrap 95% CIs (n=2000 over 1,000 queries): tq-pro 0.9994 [.999, 1.000] vs OPQ 0.9995 [.999, 1.000] — a tie, not a win; vs RaBitQ 0.9646 [.961, .969] — non-overlapping. Build cost: ~20× below OPQ here, ~4× at 1M; RaBitQ builds in 0.3 s, so the build-time advantage is over OPQ only. Holds at 1M scale (0.989 +rerank, tying OPQ, on Gutenberg LaBSE regenerable via benchmarks/gutenberg_embed.py). Full tables — the 15-method BGE-M3 comparison, the rerank frontier, KV-cache generation quality & memory, the RaBitQ estimator-isolated head-to-head — are in docs/benchmarks/embeddings.md and docs/benchmarks/kv.md. Reproduce end-to-end on public data: notebooks/turboquant_benchmark.ipynb · Colab.

The RaBitQ clause does not generalize unscoped. On six public arms, preregistered and scored by rule (benchmarks/RESULTS_rabitq_public.md; every method at matched stored bytes, 3 seeds, 5× rerank), the registered verdicts are beats RaBitQ MIXED and ties OPQ MIXED. Per arm, wins / ties / losses of the registered pipeline at rr5:

On the fixed kernel (supplementary tqfix) the full-dimension losses become ties and ties OPQ reaches HOLDS; tq-pro's own residual IVF (tq_ivf) wins 23 of 94 pairs with losses in the same two places. 21 of 540 registered cells never ran and are recorded as not run, not imputed (Amendment 4 of the preregistration).

Reading compression ratios. Ratios vary with source dimension, PCA truncation, code width, retained metadata, and whether exact originals are kept for reranking — so distinguish compressed payload vs all-in index storage vs full retrieval-pipeline storage. The canonical headline is 32× at recall@10 ≈ 0.999 above (status reported, private data); other figures in the benchmark docs (e.g. 27.7× single-vector, 114× pipeline-storage) are labeled by their accounting basis and are likewise reported rows in claims.yaml.

At scale & in production

Larger-than-RAM search (1.9.0). TQEIndex persists an index and memory-maps it; a block-streamed path keeps peak RAM at O(n_queries × block) at any corpus size. ShardedIndex splits a corpus into shards that share one PCA basis (scores stay comparable) behind a JSON manifest and fans search across them (parallel across cores; distributed.py partitions shards across machines). On disk, index format v3 bit-packs sub-byte codes — a lossless re-encoding (rankings bit-identical to v2) at 24.1 B/row vs 41 B/row in v2 (2M rows / 4-bit / --no-originals).

from turboquant_pro import TQEIndex, ShardedIndex

idx = TQEIndex.open("index.tqe", mmap=True)                 # memory-mapped, read/search only
ids, scores = idx.search(queries, k=10, block=100_000)      # bounded-RAM, block-streamed

ShardedIndex.create(corpus, "shards/", shard_size=500_000, bits=3)   # one shared PCA basis
ids, scores = ShardedIndex.open("shards/manifest.json").search(queries, k=10)

The tqp CLI covers the whole lifecycle — trace → plan → compress → certify → verify → replay → monitor, plus a persisted-index workflow:

tqp plan embeddings --embeddings corpus.npy --target "recall@10 >= 0.90"    # recipe on the Pareto frontier
tqp certify --original corpus.npy --reconstructed corpus_q.npy --min-tau 0.8 \
  --task "recall@10 >= 0.995" --environment --html report.html              # rank floor + provenance envelope
tqp verify certificate.json --original corpus.npy --reconstructed corpus_q.npy   # a third party re-checks it
tqp index create --embeddings corpus.npy --out shards/ --bits 3 --shard-size 500000
tqp index search shards/manifest.json --queries q.npy --k 10 --mmap --block 100000
tqp query "SELECT id, score FROM 'x.tqe' ORDER BY COSINE(:q) LIMIT 10 WITH (RECALL >= 0.95)" \
  --queries q.npy                                       # declare the target; the planner meets it (1.9.1)
tqp anatomy --npy corpus.npy --k 10                     # hub anatomy: what your hubs ARE (1.9.1)
tqp hubdiff --original corpus.npy --reconstructed corpus_q.npy --min-anti-recall 0.9 \
                                                        # the tail mean recall hides (1.9.1)
tqp anatomy --npy corpus.npy --strata kmeans:8 --save-map map.json   # per-stratum, not pooled (2.0.0a3)
tqp hubdiff --original corpus.npy --reconstructed corpus_q.npy --labels lang.txt \
  --abstain-fails                                       # min-over-strata; ABSTAIN (2.0.0a3)

New to hubness and anti-hubs? docs/HUBNESS_PRIMER.md — the ten-minute primer on why aggregate recall can stay green while your hardest queries collapse, and how anatomy/hubdiff catch it. Trust the tail, not the mean.

Stratified instruments (STRATA Phase 1, 2.0.0a3). A pooled hubness number can stay green while one stratum fails, so the gates run per stratum and report the minimum, never the average. Strata come from k-means, a saved area map, or a label file. Area maps are content-addressed (tqp-area-map/1): an incomplete profile matches nothing, including itself, and a tampered artifact refuses to load. A stratum with too few rows returns ABSTAIN with a registered cause rather than a pass, and --abstain-fails makes that an error in CI. The relational surface (attach_strata) exposes the same results to DuckDB for querying.

Full command reference: docs/CLI.md. Also here: QualityMonitor (cosine + (A2) tangential drift, Prometheus metrics), behavioral_agreement (decision-level flip rate + noise floor), hardware-aware profiles (Volta→Blackwell), a portable Triton fused-decode kernel, and cross-framework export (FAISS / Milvus / Qdrant / Weaviate / Pinecone) — see Integrations.

Agents & tool use

Autonomous systems can consume the whole pipeline as tools. turboquant_pro.agent_tools is a small JSON-in/JSON-out surface with docstrings written for tool-calling models — wrapped for LangChain, DSPy, an MCP server, and custom-GPT Actions in examples/agentic/.

from turboquant_pro import best_compression_at_recall, certify_ranking

plan = best_compression_at_recall(corpus, k=10, min_recall=0.99)   # "best ratio at 0.99 recall" — accepts on recall, not cosine
cert = certify_ranking(corpus, reconstructed)                      # the distribution-free rank receipt

The goal is a runtime input: the agent declares the target recall (or the consumer metric, or k) per task, and the tool accepts and certifies against that goal — never reconstruction cosine. That is the project's one rule expressed as an API, and it is why cosine can't be the gate: the coordinate worth keeping is the one that carries the currently-declared goal's geometry. Full guide: examples/agentic/README.md.

Feature & stability matrix

The full table is in docs/api-stability.md (the source of truth); component reference in docs/API.md.

Scope & honesty: results are strongest on text embeddings and LLM workloads; multimodal APIs/presets exist but are less validated. The RaBitQ comparison is the scoped, preregistered one above (six public arms, verdicts MIXED), not an unscoped "beats"; "robust across every architecture" means every architecture tested. A recorded long-generation degradation of asym-NF4 KV quant did not survive re-validation (erratum 2026-08-15: measured nf4a gap −0.31 vs the recorded 13.7 on gov_report-512, n=40; a larger real collapse, 26.64, exists under symmetric NF4 only — see benchmarks/kvquant_matrix/REVAL-2026-08-08.md and the CHANGELOG Errata); readscope's C-11c measured operator drift along the sequence — a key compressed against an early operator is later read by a different one, with a 225%-of-uniform mispricing cost, sixteen head-cells against a paired null — and it has now been run against the real degradation curve and refuted as its mechanism (amended C-12, 2026-08-15, on the symmetric-NF4 collapse that does reproduce): teacher forcing removes the consistent growth (sign test p = 0.42) and the error's orientation does no work against a rotated null, so the collapse is autoregressive compounding of a large constant error, not drift. C-11c's drift stands as a measurement; its claim to explain this degradation is dead, recorded in calibration/records/c12-longgen-drift-sym.json. Negative results and caveats are kept first-class in docs/claims.md and the soundness audit.

Not to be confused with the similarly-named turboquant (the HuggingFace KV-cache implementation of the original ICLR TurboQuant algorithm). TurboQuant Pro is a broader, retrieval-first platform that uses that quantizer as one component.

Documentation & reproducibility

• Documentation hub — guides, reference, and the 15-minute reviewer path.
• Agents & MCP: examples/agentic/ — LangChain / DSPy / MCP / custom-GPT wrappers over turboquant_pro.agent_tools.
• Artifact hashing (2026-08-16): every benchmark shard writes a config.<shard>.json sidecar whose artifact_sha256 covers the effective config, the codebook level tables, and the source of every quantizer function; the aggregator names what it actually scored and refuses single-arm reporting when hashes disagree. An arm is its tables and its arithmetic, not its label — see the 2026-08-15 erratum for why this exists.
• Reproduce the claims: CLAIMS.md (claim → notebook → hardware → status) · claim replay guide · evidence ladder.
• Benchmarks: embeddings · KV cache · release/library growth.
• Formats: FORMATS.md (TQE1 / TQIX / certificates at a glance) · FORMAT_SPEC.md · CERTIFICATE_SPEC.md.
• Extending it: two plugin points, same shape. turboquant_pro/plugins.py registers quantizers (what gets certified) — see plugins/ for tqp-bnb, tqp-gptq-awq, tqp-trtllm. turboquant_pro/read_operators.py registers read operators (what they are certified against) — see plugins/tqp-readscope/. Each has its own conformance kit, and in-tree entries register through the same public interface an out-of-tree package uses. Design: docs/DESIGN_hardware_and_plugins.md.
• Citation: CITATION.cff (GitHub "Cite this repository") · full BibTeX + acknowledgments in docs/CITATION.md.
• Contributing: CI runs ruff check and black --check over turboquant_pro/, tests/, and benchmarks/, and both must pass. Install the hooks so a mismatched local formatter version cannot redden CI: pip install pre-commit && pre-commit install. The pinned versions live in .pre-commit-config.yaml and must stay equal to the dev extras in pyproject.toml.

License

MIT License. See LICENSE. Author: Andrew H. Bond, San Jose State University.