simlens 0.2.0

Faithful, vector-only similarity & ranking attribution — the 'why' vector search never returns.

SimLens

See why your vectors match.
Faithful, vector-only similarity & ranking attribution — for any embedder, any vector store.

---

Vector search powers your RAG pipeline, your recommendations, your semantic search, your anomaly detection. And every one of them answers a question with a black box: this matches — score 0.83. Which concepts drove the match? Why did the wrong result rank first? Why was the right one buried at #7? The score won't say.

SimLens turns that score into an answer — the specific, named concepts that produced it, in plain language, and it proves the breakdown adds up. It reads only the vectors, so it drops into any stack without touching your model or your database.

Think of it as SHAP for vector search: the missing explanation layer for similarity and ranking.

import simlens

# Zero setup — an exact explanation on the very first call, no training required.
ex = simlens.Explainer(metric="cosine")
attr = ex.explain(query_vec, candidate_vec)

print(attr.as_sentence())
# → "Matched mainly on 'financial-regulation' (61%), '2024-filings' (22%)."

…or skip the manual setup entirely — point autofit at your vector store and it samples vectors, trains the dictionary, and names features from your payload fields + an optional LLM, all in one line:

bundle = simlens.autofit(store)                       # zero manual labeling
# or: simlens.autofit(store, namer=simlens.naming.from_provider("openai"))

New here? docs/how-simlens-works.md is a friendly, plain-language tour of everything below.

Why teams reach for SimLens

• Answers the real questions. Not just why does A match? but why did A outrank B? — the margin, decomposed. The question search and recsys teams actually debug.
• Faithful by construction. For dot & cosine, the explanation is the arithmetic: contributions provably sum to the score. Every result ships a completeness residual so you can see exactly how exact it is — no hand-wavy storytelling.
• Three levels of zoom. Raw dimensions (exact) → learned monosemantic features → your own named concepts — one consistent, additive contract across all three.
• Steerable. "More like this, but less of that" — edit a query in concept space and search again.
• Drop-in & fast. Embedder-agnostic, database-agnostic, and backed by a compact Rust core for the per-query hot path.
• Auditable. Every explanation is stamped with a content hash of the artifacts that produced it — reproducible rationale for decisions that have to hold up.

What you can do

Capability	Call
Explain a match (dims / features / concepts / aspects)	ex.explain(q, c, level=...)
Explain a ranking — why A beat B	ex.explain_margin(q, better, worse)
Centered "why" — discriminative, past the anisotropy baseline	ex.explain(q, c, center=True)
Any learned metric (cross-encoder, reranker) via Integrated Gradients	simlens.LearnedMetricExplainer(scorer).explain(q, c)
Why they're not more similar	ex.explain_dissimilarity(q, c)
Minimal reason — what breaks the match	ex.ablate(q, c, threshold=...)
Steer a query in concept space	ex.steer(q, {"topic": -1.0})
Contrast against a background set	ex.explain_vs_corpus(q, c, foil)
Certify a bundle's faithfulness (signed scorecard)	bundle.certify(vectors)
Summarize a whole result page	ex.summarize(q, hits)
Late-interaction (multi-vector) attribution	simlens.MultiVectorExplainer().explain(Q, C)
Auto-build a bundle from your store (no manual labeling)	simlens.autofit(store)
Train & package a concept bundle manually	simlens.train.build_bundle(...)
Calibrate / audit naming confidence	simlens.eval.reliability(bundle, X, labelers)
Fetch candidates from your store	simlens.adapters.Qdrant / Pgvector / Faiss / Weaviate
Inspect, verify, evaluate from the shell	simlens info | verify | eval | serve
Serve over HTTP	python -m simlens.serve --bundle b.simlens

System extensions

Thin, customizable wrappers that adapt SimLens to a system type — you bring the vectors, they bring the explanation. Business-logic-agnostic and configurable.

from simlens.integrations.rag   import RagExplainer          # why retrieved / why ranked
from simlens.integrations.recsys import RecsysExplainer       # "because you liked …", steer
from simlens.integrations.kg    import KnowledgeGraphExplainer # explain / propose / type edges
from simlens.integrations.audit import AuditLog               # signed, hashed decision records

Optional extras (same package): pip install "simlens[qdrant,openai,train]" — stores (qdrant, pgvector, faiss, weaviate), LLM naming providers (openai, gemini), research-scale GPU training + safetensors import (train), and sublinear kNN edge proposal (kg).

Install / build from source

python -m venv .venv && . .venv/bin/activate
pip install maturin numpy pytest
maturin develop --release      # builds the Rust core into your environment
cargo test -p simlens-core     # Rust unit tests
pytest -q                      # Python tests
python examples/quickstart.py  # full end-to-end demo

Project layout

Path	What
crates/simlens-core	Rust attribution kernels — the math and the hot path
crates/simlens-py	PyO3 bindings → simlens._native
python/simlens	Python API: Explainer, Bundle, train, eval, adapters, viz, serve
examples/	Runnable, self-contained demo
docs/	How SimLens works

Status

v0.2 — trustworthy defaults, measured not asserted. Everything in v0.1 plus:

• Real SAE trainers — TopK (default), BatchTopK, JumpReLU with AuxK dead-feature revival, a zero-dep numpy backend and an optional torch/GPU backend; the training-time sparsity gate is reproduced at inference (train == inference), and safetensors/SAELens dictionaries drop in via import_safetensors_sae.
• Detection-scored naming — names carry a measured balanced-accuracy (name-as-classifier), and low-scoring names are dropped instead of shown as confident nonsense.
• Centered "why" — anisotropy correction (mean / ABTT / whitening) surfaces discriminative concepts past the global-mean baseline; the default for the integrations.
• Integrated Gradients — explain any learned/non-linear scorer with a completeness axiom.
• Faithfulness certification — a signed quality scorecard (FVU, L0, dead %, deletion/ insertion AUC, detection accuracies) baked into the bundle and covered by its content hash.
• Zero-copy numpy FFI, criterion + Python benchmarks (docs/benchmarks.md), CI + abi3 wheels + PyPI/crates.io publishing, property-based & adversarial tests, and per-extension real-system validation.

Frontier (post-v0.2): a native Rust simlens-serve, cross-modal and hierarchical concepts.

Documentation

• How SimLens works — the concepts, end to end.
• Benchmarks · Real-system validation.
• LICENSE — Apache-2.0.

License

SimLens is released under the Apache License 2.0.