Isotrieve
Migration CI for vector stores.
Switch embedding models without re-embedding your corpus — and know, before you cut over, exactly what it costs you in retrieval quality.
pip install isotrieve
Nobody upgrades their embedding model
Not because they don't want to. Because the upgrade path is a migration project.
A better model ships. You want it. Then you price it out: re-embed the entire corpus, re-index, re-tune every downstream threshold, and pray retrieval quality survives. For a corpus of any real size, that's a quarter of engineering time and a compute bill, with no way to know whether it was worth it until it's already done.
So the model in production is the model you picked when you started. Everyone knows it's not the best one anymore. Nobody has a safe way to change it.
The problem was never the transform. It was the risk.
Techniques for mapping one embedding space onto another have existed in the literature for years. What hasn't existed is the thing that makes a migration shippable: a way to measure what you're about to lose, and a way to stop the migration if the answer is "too much."
That's Isotrieve.
How it works
1. Calibrate — Embed a small sample of your corpus with both models. Hundreds of documents, not millions. No full-corpus embedding bill.
2. Fit — Learn a transform between the two spaces: Orthogonal Procrustes, ridge/affine, or a small residual model. Cross-dimension is supported (384 → 1536 works).
3. Transform — Apply the mapping to your stored vectors, in place, in your store. Your source text is never touched. The new model is never called on the full corpus.
4. Gate — Measure retrieval retention (Recall@k, MRR) against a held-out query set. Clears your threshold, it ships. Doesn't, it fails loudly with a report explaining why — and your index is never left half-migrated.
Step 4 is the product. Steps 1–3 are how it earns the right to exist.
Quickstart
# Fit a mapping from calibration vectors
isotrieve calibrate \
--source old_vectors.npy \
--target new_vectors.npy \
--method ridge \
--out mapping.isotrieve
# Check retention before you touch anything
isotrieve gate \
--mapping mapping.isotrieve \
--queries held_out_queries.npy \
--threshold 0.95 \
--format json
# Exit code 0 = safe to migrate. Exit code 1 = don't.
Wire that gate into CI and an embedding upgrade becomes a pull request instead of a project.
Full walkthroughs live in notebooks/ — six self-contained demos, Colab-ready, including cross-architecture dimension changes and drift recalibration.
Vector store support
Isotrieve separates the transform (store-agnostic) from the adapter layer (store-specific), so one learned mapping applies to whatever you're actually running.
| Store | Query | Migrate in place | Dry run | CI-tested |
|---|---|---|---|---|
| ChromaDB | ✅ IsotrieveChromaFunction |
✅ migrate_collection() |
✅ | ✅ |
| Qdrant | ✅ QdrantAdapter.query() |
✅ QdrantAdapter.migrate() |
✅ | ✅ |
| pgvector | ✅ PgvectorAdapter.query() |
✅ PgvectorAdapter.migrate() (shadow-column swap) |
✅ | ✅ |
| LlamaIndex | ✅ isotrieve.wrappers.llamaindex |
✅ migrate_llamaindex_store() |
✅ | ✅ |
| Pinecone | ✅ PineconeAdapter.query() |
✅ PineconeAdapter.migrate() (shadow namespace) |
✅ | — (needs API key) |
| LangChain | ✅ IsotrieveEmbeddings |
— | — | ✅ |
| Weaviate | hook only | — | — | — |
| FAISS | example only | — | — | — |
Capability flags are generated from the adapter classes themselves — see docs/adapters.md.
How this differs from embedding adapter libraries
Several libraries translate between embedding spaces using pretrained, general-purpose adapters for popular model pairs. They're good at what they do. They solve a different problem.
| Adapter libraries | Isotrieve | |
|---|---|---|
| Mapping source | Pretrained on general data | Fit on your corpus |
| Where vectors live | Translated at query time | Migrated in place, in your store |
| Failure mode | You find out in production | Gate fails the build |
| Domain corpora | General-purpose fit | Calibrated to your distribution |
If you want to query an existing index with a different model today, use an adapter library. If you want to permanently move your index to a new model and be able to prove it didn't degrade, that's this.
Claims and benchmarks
Every quantitative claim in this repo links to a committed artifact under benchmarks/results/ and is listed in CLAIMS.md. Our claims linter validates the artifact links in CI (currently warn-only; legacy artifacts predate full config declarations).
Retention numbers are corpus- and model-pair specific. Yours will differ from ours. That's what the gate is for.
Adapter comparison (SciFact, MiniLM→bge-large, K=4000, 3 seeds)
| Adapter | nDCG@10 retention | Notes |
|---|---|---|
| Ridge | 0.871 ± 0.006 | Default. Fast, stable. |
| LowRank | 0.857 ± 0.009 | Compressed matrix. ~1% worse. |
| MLP | 0.727 ± 0.007 | No tuning. Linear wins. |
K-sweep (all adapters averaged, SciFact, 3 seeds)
| K | nDCG@10 retention | Gate |
|---|---|---|
| 500 | 0.671 ± 0.041 | WARN |
| 1000 | 0.735 ± 0.058 | WARN |
| 2000 | 0.785 ± 0.052 | PASS |
| 4000 | 0.832 ± 0.061 | PASS |
Same-dim pair (bge-large→e5-large, 1024→1024)
| Metric | Value |
|---|---|
| Floor (raw cross-space) | 0.0 |
| Isotrieve (mapped) | 0.667 |
| Ceiling (full re-embed) | 0.722 |
| Retention | 0.923 ± 0.010 |
Same dimension ≠ same space. e5 models require "query: "/"passage: " prefixes; without them ceiling drops to 0.36.
When NOT to use Isotrieve
- Maximum retrieval quality matters more than cost → re-embed
- Calibration domain mismatches corpus (e.g., code index calibrated on prose)
- Quality gate returns FAIL → do not migrate; re-embed
- You need unsupervised migration (Isotrieve requires paired calibration)
- K < 2000 (quality degrades significantly below this)
Anti-patterns
- Do not mix vectors from different models in one collection
- Do not assume same dimensionality means compatibility
- Do not skip the quality gate
- Do not use MLP adapter (0.727 vs 0.871 for Ridge, same cost)
Mapping types
| Type | Use case | Inverse | Save/Load |
|---|---|---|---|
| RidgeMapping | Default. Unequal dims, noise robust. | No | ✅ |
| OrthogonalProcrustesMapping | Square dims. Best when source/target are similar spaces. | Yes | ✅ |
| ProcrustesDiagMapping | Square dims. Axis-aligned transform. | Yes | ✅ |
| LowRankAffineMapping | High-dim with limited calibration data. | No | ✅ |
| ResidualMLPMapping | Non-linear transforms (requires torch). Beta. | Yes | ✅ |
CLI commands
| Command | What it does |
|---|---|
isotrieve plan |
Estimate cost: API calls, storage, time |
isotrieve calibrate |
Fit mapping from calibration vectors |
isotrieve calibrate --queries-only |
Fit mapping from query-side calibration only |
isotrieve transform |
Transform stored vectors to new space |
isotrieve gate |
Evaluate retrieval quality (PASS/WARN/FAIL) |
isotrieve inspect |
Show mapping metadata and validation report |
isotrieve report |
Render migration report as markdown |
isotrieve doctor |
Check environment and dependencies |
isotrieve manifest |
List/show migration manifests |
isotrieve rollback |
Roll back a migration (with --dry-run) |
isotrieve resume |
Resume an interrupted migration |
isotrieve verify |
Post-migration drift & gate revalidation |
isotrieve version |
Show version |
Prior art
Isotrieve builds on vec2vec, mini-vec2vec, Drift-Adapter, and the Platonic Representation Hypothesis. The contribution here is engineering — library, CLI, quality gate, store adapters, reproducible benchmarks — not algorithmic novelty. Citing the technique? Cite that work. Citing the tool? Cite this repo.
Project structure
isotrieve/
├── isotrieve-python/ # Maintained Python package (PyPI: isotrieve)
├── notebooks/ # Six self-contained demos
├── benchmarks/ # Harness and committed results
├── verification/ # Audit reports, coverage, test baselines
├── assets/ # Diagrams and visual assets
└── .github/ # CI, issue templates, gate action
Status
Actively developed, pre-1.0. APIs may change between minor versions until 1.0. Beta-marked adapters are functional but not yet load-tested.
Issues and PRs welcome — see CONTRIBUTING.md.
License
Apache-2.0. See LICENSE.
Metadata
Release files for isotrieve 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| isotrieve-0.4.0.tar.gz | 93.6 kB | With provenance | Details for isotrieve-0.4.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| isotrieve-0.4.0-py3-none-any.whl | Python 3 | none | any | Details for isotrieve-0.4.0-py3-none-any.whl |
Total release size: 211.1 kB
Release files / isotrieve-0.4.0.tar.gz
| Download URL | isotrieve-0.4.0.tar.gz |
|---|---|
| Size | 93.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c8b242923c778b0eec1b8a0054fe2eb45a5761949e0e47c76e92eda52b401c43
|
|
BLAKE2b-256 checksum How to use checksums |
dea05bcdcf6b28b961af626d224ca832030d8e28f31830a3fa3d59ed4f43421b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 30, 2026.
Transparency logRelease files / isotrieve-0.4.0-py3-none-any.whl
| Download URL | isotrieve-0.4.0-py3-none-any.whl |
|---|---|
| Size | 117.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8d4db7b1f7f030bb1ddf837b8794f03a11603d079119ea62c9ace0f49b329339
|
|
BLAKE2b-256 checksum How to use checksums |
2176572dbe13894f5612838a7052ebf1d6c3534398dc9a7cdca3a1a6aebf75cb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 30, 2026.
Transparency logSecurity for isotrieve 0.4.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for isotrieve-0.4.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 30, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.