Isotrieve

Migration CI for vector stores.

Switch embedding models without re-embedding your corpus — and know, before you cut over, exactly what it costs you in retrieval quality.

pip install isotrieve

Nobody upgrades their embedding model

Not because they don't want to. Because the upgrade path is a migration project.

A better model ships. You want it. Then you price it out: re-embed the entire corpus, re-index, re-tune every downstream threshold, and pray retrieval quality survives. For a corpus of any real size, that's a quarter of engineering time and a compute bill, with no way to know whether it was worth it until it's already done.

So the model in production is the model you picked when you started. Everyone knows it's not the best one anymore. Nobody has a safe way to change it.

The problem was never the transform. It was the risk.

Techniques for mapping one embedding space onto another have existed in the literature for years. What hasn't existed is the thing that makes a migration shippable: a way to measure what you're about to lose, and a way to stop the migration if the answer is "too much."

That's Isotrieve.

How it works

1. Calibrate — Embed a small sample of your corpus with both models. Hundreds of documents, not millions. No full-corpus embedding bill.

2. Fit — Learn a transform between the two spaces: Orthogonal Procrustes, ridge/affine, or a small residual model. Cross-dimension is supported (384 → 1536 works).

3. Transform — Apply the mapping to your stored vectors, in place, in your store. Your source text is never touched. The new model is never called on the full corpus.

4. Gate — Measure retrieval retention (Recall@k, MRR) against a held-out query set. Clears your threshold, it ships. Doesn't, it fails loudly with a report explaining why — and your index is never left half-migrated.

Step 4 is the product. Steps 1–3 are how it earns the right to exist.

Quickstart

# Fit a mapping from calibration vectors
isotrieve calibrate \
  --source old_vectors.npy \
  --target new_vectors.npy \
  --method ridge \
  --out mapping.isotrieve

# Check retention before you touch anything
isotrieve gate \
  --mapping mapping.isotrieve \
  --queries held_out_queries.npy \
  --threshold 0.95 \
  --format json

# Exit code 0 = safe to migrate. Exit code 1 = don't.

Wire that gate into CI and an embedding upgrade becomes a pull request instead of a project.

Full walkthroughs live in notebooks/ — six self-contained demos, Colab-ready, including cross-architecture dimension changes and drift recalibration.

Vector store support

Isotrieve separates the transform (store-agnostic) from the adapter layer (store-specific), so one learned mapping applies to whatever you're actually running.

Capability flags are generated from the adapter classes themselves — see docs/adapters.md.

How this differs from embedding adapter libraries

Several libraries translate between embedding spaces using pretrained, general-purpose adapters for popular model pairs. They're good at what they do. They solve a different problem.

If you want to query an existing index with a different model today, use an adapter library. If you want to permanently move your index to a new model and be able to prove it didn't degrade, that's this.

Claims and benchmarks

Every quantitative claim in this repo links to a committed artifact under benchmarks/results/ and is listed in CLAIMS.md. Our claims linter validates the artifact links in CI (currently warn-only; legacy artifacts predate full config declarations).

Retention numbers are corpus- and model-pair specific. Yours will differ from ours. That's what the gate is for.

Adapter comparison (SciFact, MiniLM→bge-large, K=4000, 3 seeds)

K-sweep (all adapters averaged, SciFact, 3 seeds)

Same-dim pair (bge-large→e5-large, 1024→1024)

Same dimension ≠ same space. e5 models require "query: "/"passage: " prefixes; without them ceiling drops to 0.36.

When NOT to use Isotrieve

• Maximum retrieval quality matters more than cost → re-embed
• Calibration domain mismatches corpus (e.g., code index calibrated on prose)
• Quality gate returns FAIL → do not migrate; re-embed
• You need unsupervised migration (Isotrieve requires paired calibration)
• K < 2000 (quality degrades significantly below this)

Anti-patterns

• Do not mix vectors from different models in one collection
• Do not assume same dimensionality means compatibility
• Do not skip the quality gate
• Do not use MLP adapter (0.727 vs 0.871 for Ridge, same cost)

Mapping types

CLI commands

Prior art

Isotrieve builds on vec2vec, mini-vec2vec, Drift-Adapter, and the Platonic Representation Hypothesis. The contribution here is engineering — library, CLI, quality gate, store adapters, reproducible benchmarks — not algorithmic novelty. Citing the technique? Cite that work. Citing the tool? Cite this repo.

Project structure

isotrieve/
├── isotrieve-python/     # Maintained Python package (PyPI: isotrieve)
├── notebooks/            # Six self-contained demos
├── benchmarks/           # Harness and committed results
├── verification/         # Audit reports, coverage, test baselines
├── assets/               # Diagrams and visual assets
└── .github/              # CI, issue templates, gate action

Status

Actively developed, pre-1.0. APIs may change between minor versions until 1.0. Beta-marked adapters are functional but not yet load-tested.

Issues and PRs welcome — see CONTRIBUTING.md.

License

Apache-2.0. See LICENSE.