thinletter-eq

Measure a compressed query encoder against your own document index — the index stays exactly as it is, only the query side gets smaller. One command line, no repository checkout: import your corpus and its existing vectors, generate synthetic queries, compile a client with llama.cpp, evaluate it against the unchanged index with a paired interval, and read the verdict of a release rule we use ourselves.

Method and numbers: technical report · released clients: huggingface.co/thinletter · browser runtime for the vector-quantised files: @thinletterio/vqweb · Apache-2.0.

Install

pip install thinletter-eq              # numpy only: import, compile, eval, compare, report
pip install "thinletter-eq[torch]"     # + torch, transformers, sentence-transformers: synth, cache

eq compile and eq eval --clients call the llama.cpp binaries (llama-imatrix, llama-quantize, llama-embedding, build b81c99b or later): put them on PATH or pass --llama-bin <dir> / --bin <file>. eq doctor tells you what is missing for which step.

The recipe

eq import  --name mycorpus --docs corpus.jsonl --doc-emb index.f32 --doc-ids ids.json --teacher qwen3-0.6b --language cs
eq synth   --dataset mycorpus --lang Czech --n-per-doc 2 --as-calib          # queries + calibration text, no real queries needed
eq cache   --dataset mycorpus --teacher qwen3-0.6b                           # fp32 query vectors of the unchanged model
eq compile --model qwen3-embedding-0.6b-f16.gguf --types Q4_K_M Q3_K --calib data/calib/mycorpus_synth_q.txt --table q4_0
eq eval    --dataset mycorpus --teacher qwen3-0.6b --clients models/*.gguf --bin llama-embedding --rule
eq compare results/perq/mycorpus_<a>.npz results/perq/mycorpus_<b>.npz     # paired bootstrap over the same queries
eq report  --out clients.md

corpus.jsonl has one {"id", "title", "text"} per line; index.f32 (or .npy) holds your document vectors in the row order of ids.json. They are never re-encoded: they are the index the client is measured against. With query vectors too (--queries, --query-emb) the fp32 model's top-10 is written as pseudo-relevance, labelled as such — then nDCG@10 means "agreement with the fp32 ranking". Without judgements the harness reports cosine to the fp32 query vector and top-10 overlap, which is what an unlabelled corpus can support.

A client that runs elsewhere (the WebGPU runtime, a service): eq queries --dataset mycorpus --texts prints what to encode, in order; eq eval --client-emb vectors.npy --client-name myclient scores the result like any other client.

What the numbers mean

eq eval reports nDCG@10 and recall@100 (NaN without judgements), cosine to the fp32 query vector and top-10 overlap with the fp32 ranking, and writes one per-query file per client. eq compare draws a paired bootstrap interval over queries (10 000 draws, seed 0) between two such files: same queries, one thing changed. Read it against two variances the interval does not contain — the calibration draw (about 0.01 nDCG@10 between two equally good quantisations) and the machine (about 0.006) — so differences under 0.01 are ties.

Release rule (--rule): keep the smallest client with nDCG@10 at least 95 % of fp32, the interval of the difference above −0.02, cosine at least 0.94 and top-10 overlap at least 0.75. What we found with it (report §2–3): at 3.4 bits a 0.6B encoder keeps 96–101 % of its retrieval quality on four English corpora; at low bit widths the language of the calibration text matters more than its domain; tolerance to compression is a property of the checkpoint (one 0.6B model holds 99 % at 3.4 bits, another of the same architecture fails below 4.5); llama.cpp's own quantiser with an importance matrix is as good as anything fancier at 200 MiB, which is why eq compile is exactly that.

Teachers known to the package (--teacher): qwen3-0.6b (Qwen/Qwen3-Embedding-0.6B), harrier-0.6b (microsoft/harrier-oss-v1-0.6b), bge-m3 (BAAI/bge-m3), me5-small, qwen3-4b, e5-mistral-7b, and the other keys of eq.teacher.TEACHERS; each carries the query prompt, pooling and context length the evaluation must reproduce.

Not in this package

The vector-quantised pipeline (2–3.6 bits per weight, the .vqw container, GPTQ onto the K-quant grid) lives in the repository as scripts; it pulls in research modules that are not ready to be a library. Its released files run in the browser through @thinletterio/vqweb, and eq eval --client-emb measures them like any other client.

Licence, contact

Apache-2.0. Jan Rosecký, thinletter — info@thinletter.io.