Agents build RAG pipelines, semantic search and recommendation retrieval every day, but they cannot see the embedding space they are searching. When retrieval goes wrong it rarely looks like a bug. It looks like "the same three chunks come back for everything", "every score is 0.8-something", "top-k is full of copies", or "brute force finds it and HNSW does not".

Each of those failures has a known, deterministic diagnostic in the literature. vecscope turns them into one command and one JSON: measurements, ok / warn / fail verdicts, and an explanation an agent can act on. It runs locally on NumPy, needs no model or network, and reports the same numbers every time.

vecscope diagnoses the vector search implementation: the geometry the model produced, the hygiene of the corpus, the distance metric, the index and the query pipeline. It does not measure relevance. Whether the right documents come back for a question needs labeled queries, and a clean audit is evidence that search is not sabotaging retrieval, not proof that retrieval is good.

Quick start

1. Install

pip install vecscope

2. Get a file of embeddings

vecscope reads a file with one embedding vector per row: the (n, dim) array a model's encode returns, saved as .npy, .parquet, .jsonl, .json or .csv (all formats).

import numpy as np
embeddings = model.encode(texts)          # shape (n, dim), e.g. (3000, 128)
np.save("embeddings.npy", embeddings)

No embeddings at hand? Generate three synthetic matrices (healthy, anisotropic, collapsed):

python -c "import urllib.request as u; u.urlretrieve('https://raw.githubusercontent.com/076923/vecscope/main/examples/make_examples.py', 'make_examples.py')"
python make_examples.py

3. Run the audit

vecscope audit embeddings.npy

vecscope 0.4.0  n=3000 dim=128 k=10  verdict=FAIL
  [ok  ] hubness                skewness=1.1661  (ok < 2, warn >= 2, fail >= 4)
                               k-occurrence skewness 1.17; top 1% of items take 3.9% of all neighbor slots (max N_k=51, expected 10).
  [WARN] antihubs               antihub_excess=0.0523  (ok < 0.05, warn >= 0.05, fail >= 0.2)
                               5.2% of items are never anyone's neighbor (random data at k=10 would give 0.0%); the excess is unreachable through k-NN retrieval.
                               try: inspect a sample of never-retrieved items; drop empty inputs, then apply the hubness fix
  [FAIL] anisotropy             mean_random_cosine=0.8732  (ok < 0.5, warn >= 0.5, fail >= 0.85)
                               random pairs have mean cosine 0.873 (isotropic ~0); mean unit vector has norm 0.93; centering would bring the cosine to -0.000.
                               try: subtract the corpus mean vector and re-normalize, applying the same transform to queries; details.mean_random_cosine_after_centering shows the expected result
  [ok  ] effective_rank         effective_rank_ratio=0.5407  (ok >= 0.1, warn < 0.1, fail < 0.05)
                               effective rank 69.2 of 128 attainable (128 dims; 54.1%); low values mean most dimensions carry nothing.
  [ok  ] collapse               near_duplicate_ratio=0.0000  (ok < 0.02, warn >= 0.02, fail >= 0.1)
                               0.0% of items have a neighbor with cosine >= 0.99 or an exact copy (exact duplicates 0.0%); no row has a near-duplicate.
  [ok  ] rogue_dimensions       top1_dim_variance_ratio=1.3070  (ok < 10, warn >= 10, fail >= 50)
                               dimension 94 alone holds 1.0% of variance, 1.3x the uniform share of 0.78%; top 3 dims hold 3.0%. Rogue dimensions dominate cosine.
  [ok  ] norms                  norm_cv=0.0000  (ok < 0.1, warn >= 0.1, fail >= 0.3)
                               vector norms vary with CV 0.000 (median 1.000, range 1.000-1.000); 0.00% are zero vectors. Unequal norms bias inner-product search toward long vectors.
  [ok  ] metric_mismatch        topk_overlap=1.0000  (ok >= 0.9, warn < 0.9, fail < 0.7)
                               cosine and inner-product top-10 share 100.0% of neighbors; 100.0% of items get identical lists. Low overlap means the index metric changes the results.
  [ok  ] distance_concentration contrast_over_noise=2.2784  (ok >= 1.4, warn < 1.4, fail < 1.1)
                               on 3000 rows a random item is 3.27x farther than the nearest neighbor (random distance 0.127); isotropic noise at 128-d and the same rows would give 1.44x, so the contrast is 2.28x that of noise. On all rows the contrast is 3.27x (median NN distance 0.039). Near 1.0 means distances carry no signal.
  [ok  ] outliers               outlier_ratio=0.0030  (ok < 0.01, warn >= 0.01, fail >= 0.05)
                               0.30% of items have no neighbor within cosine distance 0.050 (median 0.039); see details.sample_ids. Typically junk, empty or foreign-language content.
  [ok  ] connectivity           island_ratio=0.0000  (ok < 0.01, warn >= 0.01, fail >= 0.1)
                               0.00% of items sit in 0 small island(s) of the k-NN graph (fewer than 15 items; 60 components in total). Graph-based ANN (HNSW) attaches islands to the wrong region or never reaches them.

4. Read the result

• The first line gives the overall verdict: the worst verdict among the findings.
• Each finding shows the measured value, the range that would be ok, and a sentence saying what the number means for retrieval. Findings that are not ok add a try: line with the smallest change to test first.
• The exit code is 1 on FAIL, so the same command works as a CI gate.
• Add --json to get the same evidence as JSON for an agent or a script.

What vecscope measures

Every finding in the JSON carries its healthy_range, warn_at and fail_at, plus a details block with the raw numbers. For anisotropy that block includes the cosine the same vectors would have after centering, so the agent can see whether centering alone fixes the problem. ann_recall appears only when you pass an ANN neighbor file with --ann; ood_queries and query_hubness only when you pass query embeddings with --queries.

Each finding also carries likely_causes, suggested_action and verify_by. They are hypotheses, not prescriptions: apply the smallest change, re-audit, and compare retrieval on real queries before adopting it. The agent skill spells out that loop.

Thresholds are defaults calibrated on seven public models and re-checked against 19 models on three labeled datasets (verification); they are a starting point for diagnosis, not a universal quality bar. Embedding spaces differ by model, language, domain and dimensionality, so a warn on a good model is normal and a clean audit does not guarantee good search. Override any pair per run:

vecscope audit vectors.npy --thresholds my_thresholds.json
# my_thresholds.json: {"anisotropy.mean_random_cosine": [0.4, 0.7], "hubness.skewness": [2.5, 5]}

The keys are listed in src/vecscope/evidence.py (THRESHOLDS) and echoed in every JSON result under thresholds.

Queries and clusters

Pass the query embeddings too and vecscope checks whether queries live where the corpus lives. Asymmetric models used without their query prefix, a different model on the query side, or questions about topics the corpus does not cover all show up as ood_queries; the same few chunks answering everything shows up as query_hubness.

vecscope audit corpus.npy --queries queries.npy

Add --clusters 10 to append a per-cluster map: size, tightness, near-duplicate share, how much of the hub traffic each cluster takes, and sample ids the agent can look up. It carries no verdicts; it tells the agent where a problem lives.

  clusters (3):
     id   size  share  cos2ctr  neardup  hubshare  sample_ids
      2   1113  37.1%    0.217     0.0%     37.0%  [1282, 1611, 496, 283, 2041]
      0   1023  34.1%    0.232     0.0%     34.2%  [1855, 2932, 1786, 1025, 2476]
      1    864  28.8%    0.222     0.0%     28.9%  [2505, 647, 2740, 2499, 2965]

Index sweep

vecscope sweep builds HNSW and IVF indexes with a small parameter grid and reports, for each setting, recall@k against exact k-NN, build time, queries per second, single-query latency percentiles (p50/p95/p99) and the serialized index size on this corpus. It then recommends the fastest setting that reaches the target recall (default 0.95) and reports its margin over the target. A margin under 0.02 marks the recommendation tight and names the fastest setting with real headroom, because recall at fixed parameters falls as the corpus grows: a setting that just clears the target on a 50k-row sample usually misses it on the full corpus. It needs the faiss-cpu extra.

pip install "vecscope[index]"
vecscope sweep embeddings.npy -k 10 --target-recall 0.95

  index params                                        recall  build s       qps  p50 ms  p95 ms     MiB  dev
 *hnsw  M=8, ef_construction=100, ef_search=16         0.987     0.35    237954   0.034   0.051     0.6  cpu
 *hnsw  M=16, ef_construction=100, ef_search=32        0.998     1.23    165139   0.084   0.108     0.8  cpu
 *ivf   nlist=76, nprobe=64                            0.999     0.14     33251   0.087   0.095     0.4  cpu
  recommendation: hnsw (M=8, ef_construction=100, ef_search=16)  recall=0.987  qps=237954  p95=0.051ms  index=0.6MiB  margin=+0.037  (fastest configuration with recall@5 >= 0.95)
  note: recall is measured on 200 queries drawn from the corpus itself; real query traffic can differ, pass it as queries when available
  note: HNSW recall at fixed parameters falls as the corpus grows, so a sweep on a sample is optimistic for HNSW; IVF nlist is set from the row count (4*sqrt(n)), so its recall at a given nprobe can move either way with size. Re-run at production size before fixing parameters
  note: queries_per_second is the median of 3 runs on this machine with one query batch; it ranks settings against each other and is not a serving throughput
  note: latency_ms_p50/p95/p99 time 200 queries one at a time in this process, with no network, concurrency or serving stack; compare settings with them, do not quote them as service latency
  note: index_bytes is the serialized index (vectors plus graph or lists); the process needs more while building, and a product-quantized index would be smaller

Queries default to a seeded sample of the corpus; pass --queries to time real queries, and do so whenever a query log exists: the setting that is fastest at the target recall on corpus-drawn queries can differ from the one real traffic needs. Each setting is timed three times and the median kept (--timing-repeats); --latency-queries (default 200, 0 to skip) are also searched one at a time for the percentiles, and index_bytes is the size of the serialized index, so a setting can be judged on memory as well as speed. The notes at the end are part of the JSON as caveats. The MCP tool is sweep_index, with the same options as parameters (n_queries, target_recall, device, margin, timing_repeats, latency_queries).

The sweep answers one question: which parameters reach the target recall on this corpus and these queries, and what they cost. The numbers are relative to this machine and this process; confirm the chosen setting in the serving environment with its own latency, throughput and memory measurements before fixing it.

With a GPU build of faiss (pip install faiss-gpu-cu12 on Linux) the IVF index and the exact baseline run on the GPU automatically; the output header names the device. HNSW has no GPU implementation in faiss and stays on the CPU. Force a device with --device cpu|gpu.

Compare two embeddings

When two models embedded the same items, or one model did before and after a change, vecscope compare tells how far the two spaces agree on who is near whom. It reports how often the nearest neighbor and the top-k lists coincide, the near-duplicate share in each space and in both, whether both spaces pick the same duplicate partner, and the rows that disagree most. It carries no verdicts. Its job is to separate item problems from model problems: a 10% collapse that both spaces flag with the same partners is a property of the items (the same image under several ids, say); duplicates that only one space flags point at that model.

vecscope compare clip.npy lens.npy -k 10          # rows must be the same items in the same order

vecscope 0.6.0  compare  n=3000 dim=128/128 k=10  neighbors=recomputed/full
  nn1_agreement=0.7050  topk_overlap=0.8537  topk_overlap_p10=0.7000  rows_with_no_overlap=0.0000
  near_dup A=0.0400  B=0.0400  both=0.0400  only_A=0.0000  only_B=0.0000  same_nearest_among_both=1.0000
    the two spaces share the nearest neighbor for 70.5% of rows and 85.4% of their top-10 lists
    on average (0.0% of rows share nothing). Near-duplicates: 4.0% in A, 4.0% in B, 4.0% in both,
    and 100.0% of those agree on the partner. Duplicates flagged by both spaces with the same
    partner are a property of the items; duplicates flagged by one space only point at that model.
  largest disagreements (row: nearest in A / nearest in B, top-k overlap):
    1769: 1340 / 1340  0.50
    1810: 1229 / 1229  0.50

Read neighbor_scope before quoting the agreement. Above --sample rows (default 100k) a seeded subset is compared, and by default the neighbors are recomputed exactly inside that subset: an item whose true neighbors fell outside the sample is compared on substitutes, which agree more often than the real ones. On a 2,000-row synthetic pair of spaces, a 500-row sample reports a top-10 overlap of 0.75 where the full corpus gives 0.57. Such a result is marked neighbor_scope: sample, and the explanation says so. To compare full-corpus rankings on a sample of rows, pass neighbor lists computed on the full corpus in each space, exact or from the production index (an (n x >= k) id file, self ids and negative fillers are dropped):

vecscope compare clip.npy lens.npy -k 10 --neighbors-a clip_knn.npy --neighbors-b lens_knn.npy

The result is then neighbors=supplied/full: the rows may be sampled, their neighbors are not, and the agreement is between the lists you supplied, exact or approximate as they were computed. The MCP tool is compare_embeddings (neighbors_a_path, neighbors_b_path).

Use with an agent

vecscope ships an MCP server with four read-only tools: audit_embeddings, sweep_index, compare_embeddings and describe_findings. Register it with any MCP client (Claude Code, Cursor, Codex, ...):

pip install "vecscope[mcp]"

{ "mcpServers": { "vecscope": { "command": "vecscope-mcp" } } }

Then ask:

Audit the embeddings in data/chunks.npy before we tune the retriever, and tell me
which finding to fix first.

The tools are read-only, but they read whatever path the agent names. On a machine that holds data the agent should not see, restrict the server to the directories with embeddings; any path outside them (after resolving .. and symlinks) is refused before it is opened:

{ "mcpServers": { "vecscope": { "command": "vecscope-mcp", "args": ["--allow-dir", "/data/embeddings"] } } }

--allow-dir can be repeated; the environment variable VECSCOPE_ALLOWED_DIRS (paths separated by : on Unix, ; on Windows) does the same. Without either, every file the server process can read is reachable.

The agent skill in .agents/skills/vecscope/SKILL.md tells the agent when to run the audit and what to do about each verdict: center or swap the model, deduplicate, re-rank, or rebuild the index. The server also carries those instructions, so an agent that only has the MCP connection follows the same loop. sweep_index runs the index sweep, compare_embeddings the comparison; describe_findings lists every finding with its thresholds, causes and actions. Responses start with a one-line summary; pass verbose=false to drop details for findings that are ok. Agents without MCP can run the CLI with --json and read the same evidence.

If the server stops connecting ("vecscope-mcp not found"), the Python environment it was installed into has usually been replaced; this happens on hosted notebooks and clusters that rebuild their environment. Check with pip show vecscope in the interpreter the client uses, reinstall there, and prefer registering the server through that interpreter so the error names the right environment:

{ "mcpServers": { "vecscope": { "command": "/path/to/env/bin/python", "args": ["-m", "vecscope.mcp_server"] } } }

Use as a library

import numpy as np
from vecscope import audit

embeddings = np.load("embeddings.npy")    # any (n, dim) float array

ev = audit(embeddings, k=10)
ev.verdict                                # "ok" | "warn" | "fail"
ev.findings[0].value                      # measured value of the first finding
ev.to_dict()                              # the full evidence, same as --json

Use in CI

The JSON output is stable across runs and carries schema_version, so it can be committed as a baseline and diffed on every index rebuild. --baseline prints each finding's change and exits 1 when any verdict got worse (or when --fail-on trips), so the pipeline fails on a regression and not on a warn that was already there.

vecscope audit embeddings.npy --json > baseline.json          # once, reviewed and committed
vecscope audit embeddings.npy --baseline baseline.json        # on every rebuild

  baseline baseline.json: verdict ok -> fail  (n, dim, k, sampling, thresholds or near-dup cutoff differ: values and verdicts are not directly comparable)
    hubness                1.0454 -> 0.8594  (-0.1860)  ok -> ok
    antihubs               0.0383 -> 0.0105  (-0.0278)  ok -> ok
    anisotropy             0.0105 -> 0.0127  (+0.0022)  ok -> ok
    collapse               0.0000 -> 0.8000  (+0.8000)  ok -> fail  WORSE
    ...
  regressions: collapse

The example diffs healthy.npy against collapsed.npy, which have different row counts, hence the note on the first line; a rebuild of the same corpus with the same settings prints no note. With --json the diff is attached as baseline_diff; the library call is diff_evidence(new, old). A baseline written by an older schema_version is rejected with a message to re-create it. Audit every row (--sample 0) for the baseline and the check, so the two runs see the same rows.

A verdict is a coarse signal: a value can drift toward the threshold for months without crossing it. Every change row therefore carries worsened (the value moved in the bad direction) and worsening (by how much, in the metric's units, positive = worse), and --tolerances turns that drift into a gate independent of the verdict bands:

# tolerances.json: {"collapse": 0.01, "ann_recall": 0.02, "hubness": 0.3}
vecscope audit embeddings.npy --baseline baseline.json --tolerances tolerances.json --require-comparable

A finding whose value worsens by more than its tolerance is listed under value_regressions (text: VALUE WORSE by 0.0300 > 0.01) and exits 1 even when its verdict is unchanged; findings without a tolerance are judged by verdict alone, and a value that improves is never a regression. The tolerance is a budget per rebuild against the committed baseline, so re-commit the baseline when a change was reviewed and accepted. The run also reports comparable: false when n, dim, k, sampling, thresholds or the near-duplicate cutoff differ from the baseline. --require-comparable makes that an error (exit 2) instead of a note, which is the right policy for a gate, since a diff across different settings measures the settings, not the corpus. The library call is diff_evidence(new, old, tolerances={...}).

Two different rules can fail a run, and the output keeps them apart. The absolute rule is the verdict against the thresholds (--fail-on fail by default, --fail-on warn to be strict); the baseline rules are regressions (a verdict got worse) and value_regressions (a value worsened beyond its tolerance). The defaults are calibrated on public models, so a domain whose healthy corpus sits at warn or fail on some finding should not be blocked by the absolute rule at all: --fail-on none --baseline baseline.json judges only the change against the reviewed baseline. Every run ends with a gate block (JSON) or line (text) that names the rule that decided:

  gate: FAIL; verdict fail reaches --fail-on fail; verdict worse than baseline: collapse
  gate: pass (baseline)

Treat the gate as a trend tracker first. Which tolerance is "too much" depends on how the findings relate to retrieval quality on your own queries; the retrieval quality study shows that relation for public datasets, and calibrating thresholds says how to measure it on yours before blocking merges on it.

CLI options

vecscope audit vectors.npy -k 20                       # neighbors per item (default 10)
vecscope audit vectors.parquet --json                  # JSON evidence
vecscope audit vectors.npy --ann hnsw_neighbors.npy    # score ANN recall against exact k-NN
vecscope audit vectors.npy --queries queries.npy       # add query-side findings
vecscope audit vectors.npy --clusters 10               # append a per-cluster report (at most n/10 clusters)
vecscope audit vectors.npy --thresholds th.json         # override (warn, fail) pairs
vecscope audit vectors.npy --fail-on warn              # exit 1 on WARN as well
vecscope audit vectors.npy --fail-on none --baseline old.json   # ignore the absolute thresholds; fail only on regressions
vecscope audit vectors.npy --baseline old.json         # diff against an earlier --json run; exit 1 on regression
vecscope audit vectors.npy --baseline old.json --tolerances tol.json   # also exit 1 when a value worsens beyond its tolerance
vecscope audit vectors.npy --baseline old.json --require-comparable    # exit 2 when the baseline used different settings
vecscope audit vectors.npy --near-dup 0.98             # cosine threshold for near-duplicates
vecscope audit vectors.npy --sample 0                  # audit every row (default samples 100k)
vecscope audit vectors.npy --sample-repeats 3          # re-audit 3 samples; report value ranges and verdict stability
vecscope sweep vectors.npy --target-recall 0.95        # HNSW/IVF parameter sweep (faiss extra)
vecscope sweep vectors.npy --device gpu                # IVF on the GPU (faiss GPU build)
vecscope sweep vectors.npy --margin 0.05               # headroom under which the pick is marked tight (default 0.02)
vecscope sweep vectors.npy --n-queries 2000 --timing-repeats 5   # queries to time (default 1000), timing runs per setting (default 3)
vecscope sweep vectors.npy --latency-queries 0         # skip the single-query latency percentiles (default 200 queries)
vecscope compare a.npy b.npy -k 10                     # agreement between two embeddings of the same rows
vecscope compare a.npy b.npy --neighbors-a a_knn.npy --neighbors-b b_knn.npy   # full-corpus neighbors on a sample of rows

Input formats

The embedding file holds one vector per row. Every format is converted to a 2-D float32 array of shape (n, dim) before the audit. Values must be finite: NaN, Inf and magnitudes above 1e16 (whose squares overflow float32) are rejected with a message. In the examples embeddings is that array, for instance the output of a model's encode:

embeddings = model.encode(texts)        # shape (n, dim), e.g. (3000, 128)
embeddings[0]                           # the vector of the first item
# array([ 0.0312, -0.1187,  0.0459, ...], dtype=float32)

The optional --ann file (or ann_path in MCP) is an integer array of shape (n, k): row i holds the ids of the k neighbors an ANN index returned for item i, in the same row order as the embedding file. Any of the formats above works. Ids refer to row positions, so the file must come from an index built on exactly these rows, so every row has to be audited: above the --sample size (100k rows by default) pass --sample 0, or the audit refuses. Ids that point at the item itself are ignored, so an index that returns self first should be queried with k + 1.

The optional --queries file (queries_path in MCP) holds query embeddings in any of the same formats, with the same dim as the corpus.

Performance

Exact k-NN is quadratic in the number of rows: doubling the rows roughly quadruples the time, and a full audit (--sample 0) of 100k rows takes minutes on a laptop. The k-NN pass works in row blocks sized to keep its distance matrix near 1 GiB whatever the row count; the other passes need a few float32 copies of the matrix, so the peak grows with rows x dims at roughly four times the matrix size. Measured on an 8-vCPU AMD EPYC 7R13 with 8 BLAS threads (NumPy only, single process, clustered synthetic corpus, k=10); memory is the peak the audit itself allocates, the input matrix comes on top. A three-run median in separate processes agreed with these single runs within 2%, and the OS peak memory within 0.04 GiB:

Time grows close to n² (3.5 to 3.9x per doubling from 50k up); memory is flat from 50k rows at 384 and 768 dims because the k-NN block shrinks with n, and steps up at 1,536 dims and at 200k rows. The default 100k-row sample keeps any corpus near the 100k rows. --queries adds one more pass over the corpus per query batch, and metric_mismatch adds a second k-NN pass only when vector norms vary. A sampled audit is a statement about the sample: use --sample 0 when you need every island and outlier counted. python examples/benchmark_scale.py --rows 50000 100000 --dim 768 re-measures the table on your machine (wall-clock and the peak NumPy allocates inside the audit) at the sizes you care about, and examples/benchmark_sweep.py does the same for the index sweep; both take --corpus (clustered, isotropic, anisotropic, duplicates, lowrank, zipf) and record the machine in their JSON.

Blocking bounds memory, not work: the exact pass still costs about n² × dim multiply-adds, so a million rows at 768 dims is roughly 100 times the 100k-row figure. For corpora that size, audit the default sample (or several with --sample-repeats) and reserve --sample 0 for the findings that need every row. Which findings those are:

A sampled audit therefore lists inconclusive_findings (the verdicts the table marks inconclusive), flags them INCONCLUSIVE (sampled) in the text output and names them in the gate line, takes the collapse verdict from details.estimated_full_share (the sampled share divided by the sample fraction), and repeats each caveat in the finding's explanation next to the number. The 200k-row figures come from the verification; tests/test_regression_injection.py checks the same properties at 6k rows on every run.

Every number is exact for the rows it saw; vecscope never substitutes an approximate neighbor search for the exact one. The only approximation is the choice of rows.

How vecscope works

vecscope computes exact k-NN once over the matrix and derives every diagnostic from it: the k-occurrence distribution gives hubness and anti-hubs, nearest-neighbor cosines give collapse, outliers and distance concentration, the k-NN graph gives connectivity, random pairs and the covariance spectrum give anisotropy, effective rank and rogue dimensions, vector norms give the norm spread, a second inner-product pass (skipped when norms are uniform) gives metric mismatch, and the ANN neighbor file, if present, is compared against the exact neighbors for recall. Query embeddings, if present, get their own exact search against the corpus for the query-side findings.

Everything is deterministic: fixed seeds, exact neighbors, no learned components, no network. Two runs on the same file give the same JSON. Exact k-NN is quadratic in the number of rows, so inputs above 100,000 rows are audited on a seeded random sample unless you pass --sample 0.

Real embeddings

Thresholds are calibrated on public, pre-computed embeddings from seven models over eight datasets: OpenAI ada-002 and text-embedding-3-large, Cohere embed-v3 (Korean and English Wikipedia), all-MiniLM-L6-v2, bge-m3, mxbai-embed-large-v1 and gte-base-en-v1.5. The table shows the measured value for the first 40k rows of each dataset; cells without a label are ok. python examples/real_models.py reproduces it, no model download needed.

A re-measurement of this table reproduced all 88 verdicts and corrected one printed value (Cohere ko distance_concentration, 1.73 to 1.72). The only warns are hubness on gte-base and effective_rank on ada-002 and mxbai-large. ada-002's well-known anisotropy (0.692) sat at warn under the earlier threshold of 0.5: its mean unit vector has norm 0.83, so unrelated documents start from a cosine floor near 0.7. Its successor is isotropic on the same corpus (norm 0.29). Centering brings every model here to a random-pair cosine of 0.00, which is why the finding reports that number. Hubness skewness of 1.2 to 1.9 is the normal range for text embeddings, so the warn threshold sits at 2.0; gte-base on arXiv abstracts sits just above it. The anisotropy thresholds (0.85, 0.95) come from the retrieval-quality study below.

Do the findings predict retrieval quality?

Partly, and in a specific way. docs/retrieval_quality.md measures every finding against nDCG@10 on two labeled BEIR datasets (SciFact, NFCorpus) with exact cosine search, for eight public models and for controlled defects injected into the best one. python examples/retrieval_quality.py reproduces it.

• Within one pipeline the findings track damage. Injecting a rogue dimension, cutting the rank, adding noise or duplicating rows moves the watched finding to fail (from ok, or from warn where the corpus already carried a few duplicates) while nDCG@10 falls, and the drop is monotone in the defect level. Queries embedded with the wrong model score ood_queries = 1.0.
• Across good models the geometry findings do not rank the models. The best models on both datasets (gte-base, bge, e5) have the highest anisotropy (0.58 to 0.82, counting the 19 models of the verification) and ordinary effective rank. What does separate the weak models is norms with metric_mismatch (unnormalized outputs) and hubness, at Spearman |rho| of 0.4 to 0.8 with nDCG@10.
• Anisotropy alone does not hurt ranking. Adding a shared offset until the random-pair cosine reaches 0.98 leaves nDCG@10 unchanged. It compresses scores, so absolute cosine thresholds and score calibration break; distance_concentration is the finding that moves when the signal itself is lost.

The anisotropy and distance_concentration thresholds were set from this study; the verification on 19 models and a Korean dataset then moved anisotropy, ood_queries and query_hubness. What neither covers is how well warn and fail predict a quality drop in other domains or on production query logs. Until that is measured for your domain, read a verdict as a reason to look and a baseline regression as a reason to compare retrieval, not as a quality score.

Calibrating thresholds to your domain

The defaults are calibrated on public models and public datasets; a fail on your corpus is a reason to look, not proof that retrieval is bad, and all-ok is not proof that the right documents come back. Before using a threshold or a tolerance as a blocking gate, measure what it predicts on your own data:

1. Collect labeled queries from the domain (a few hundred with relevant document ids is enough) and split them: one part to pick thresholds, a held-out part to check them.
2. Run vecscope audit --json on each candidate state of the pipeline (models, chunkers, transforms, index settings) and compute retrieval quality (nDCG@k, Recall@k, MRR) with the same queries on the same state. examples/retrieval_quality.py does both for the public datasets and can be pointed at your own.
3. Relate each finding's value to the quality metric across those states. A finding that does not move with quality in your domain should not gate it, whatever its default says.
4. Set --thresholds and --tolerances from the states where quality was acceptable, using only the first split.
5. Confirm on the held-out split that the thresholds separate the acceptable states from the rest. A threshold tuned and checked on the same queries will look better than it is.

The same procedure applies to a suggested_action: it is a hypothesis about the cause, and centering, standardizing or truncating vectors changes rankings. Keep the original vectors and retrieval results as the baseline, apply the transform to a copy, measure quality, latency and memory on both, and adopt the change only when the labeled queries improve, not when the finding turns ok.

FAQ

Documentation

• Agent skill: when to run the audit and what to do with each verdict
• Retrieval-quality study: every finding against nDCG@10 on labeled data
• Verification: exactness, recall scoring, the sampling claims, performance, and 19 models on 3 labeled datasets
• Examples: synthetic matrices, the calibration script and the retrieval-quality study
• Changelog

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for how to run the tests and what a good change looks like. Security reports go through SECURITY.md.

License

MIT