DiffVector

The open-source visual git-diff for vector embeddings — detect semantic drift and search regression before changing your embedding models or chunking strategies.

Author: shunvel · shunvel@gmail.com

Install (no git clone required)

pip — CLI + GUI

pip install diffvector

# Optional: OpenAI, Cohere, HuggingFace embedding providers
pip install "diffvector[models]"

# Launch the GUI
diffvector gui

# Run a CI audit
diffvector audit \
  --source chroma --path ./chroma_db --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.6 --json

# List embedding models
diffvector models

Docker — GUI without Python

docker run --rm -p 8501:8501 ghcr.io/shunvel/diffvector:latest

Open http://localhost:8501.

Pass API keys and mount local Chroma data:

docker run --rm -p 8501:8501 \
  -v "$(pwd)/chroma_db:/data/chroma_db" \
  -e OPENAI_API_KEY \
  -e PINECONE_API_KEY \
  ghcr.io/shunvel/diffvector:latest

Qdrant on the host (Docker Desktop):

docker run --rm -p 8501:8501 \
  --add-host=host.docker.internal:host-gateway \
  -e QDRANT_URL=http://host.docker.internal:6333 \
  ghcr.io/shunvel/diffvector:latest

Or use the included compose stack (Qdrant + GUI):

docker compose --profile gui up --build

For OpenAI/Cohere/HuggingFace models inside Docker, build with model providers (~2 GB image):

docker build --build-arg INSTALL_MODELS=1 -t diffvector:full .
docker run --rm -p 8501:8501 diffvector:full

From source (contributors)

git clone https://github.com/shunvel/diffvector.git
cd diffvector
pip install -e ".[dev]"
diffvector gui

Quick Start (local testing)

After pip install diffvector (or editable install from source).

Option A — Chroma only (simplest, no Docker)

# Seed local test data (zero API keys needed)
python -m utils.mock_generator

# Launch the GUI
diffvector gui

Open http://localhost:8501 and use the sidebar settings below (see Local Testing Guide).

Option B — Chroma + Qdrant (Docker Desktop)

# One command: start Qdrant + seed both backends (requires git clone for scripts/)
chmod +x scripts/setup-local.sh && ./scripts/setup-local.sh

# Or with pip only:
docker compose up -d
python -m utils.mock_generator --backend all

diffvector gui

Or manually:

docker compose up -d
python -m utils.mock_generator --backend all
diffvector gui

GUI settings for Qdrant:

Qdrant dashboard: http://localhost:6333/dashboard

Option C — Pinecone (cloud, needs API key)

cp .env.example .env
# Add: PINECONE_API_KEY=pcsk-...

python -m utils.mock_generator --backend pinecone --collection demo --count 100
diffvector gui

GUI settings for Pinecone:

Pinecone has no local Docker image — the mock generator creates a serverless index named demo (1536-d, cosine) and upserts 100 test vectors.

macOS Docker note: If docker is not found in a new terminal, Docker Desktop's binary may not be on your PATH:

export PATH="/Applications/Docker.app/Contents/Resources/bin:$PATH"

The included scripts/setup-local.sh handles this automatically (source install only).

Why DiffVector?

When engineering teams upgrade embedding models (e.g., OpenAI text-embedding-ada-002 → text-embedding-3-small), alter chunking strategies, or migrate across vector database vendors, they are structurally blind.

Changing models alters high-dimensional spatial distances completely. Traditional migration scripts focus on moving data pipelines, leaving developers with zero insight into search quality regression, semantic drift, or payload mismatch.

The result is Silent Retrieval Decay: your system returns mathematically valid nearest neighbors that are textually irrelevant. DiffVector catches this before production.

How It Works

DiffVector is a non-destructive diagnostic staging area. It never writes to your production index.

flowchart LR
  subgraph sources [Your Vector DB]
    Chroma
    Qdrant
    Pinecone
  end
  subgraph engine [DiffVector Engine]
    Ingest[1. Ingest Slice]
    Simulate[2. Simulate Model]
    Diagnose[3. Diagnose Drift]
  end
  sources --> Ingest
  Ingest --> Simulate
  Simulate --> Diagnose
  Diagnose --> GUI[Streamlit GUI]
  Diagnose --> CLI[CLI / CI Gate]

The 3-Step Engine

CLI (CI Gate)

Gate embedding upgrades in your pipeline:

# Audit with JSON output — exits 1 if overlap below threshold
diffvector audit \
  --source chroma \
  --path ./chroma_db \
  --collection demo \
  --target-model local/offline-hash-384 \
  --n 300 \
  --topk 5 \
  --min-overlap 0.6 \
  --json

# List supported models
diffvector models

Canary queries (optional)

Provide a source model to embed free-text queries in both spaces:

diffvector audit \
  --source chroma --path ./chroma_db --collection demo \
  --target-model text-embedding-3-small \
  --source-model text-embedding-ada-002 \
  --queries canaries.txt \
  --min-overlap 0.5

Local Testing Guide

This section covers zero-config local testing — no production vector DB, no OpenAI/Cohere keys, and no HuggingFace download required.

Which database can I use locally?

You do not need an existing production database. DiffVector ships a mock data generator:

Start with Chroma for the fastest path. Use Qdrant when you want to validate the Qdrant connector specifically. Use Pinecone when you need to verify the cloud connector before production migration.

Which test command should I run?

Run pytest during development. Run test-all-backends.sh before a release when you want all three connectors verified in one shot. Run test-cli.sh for the full pre-publish gate (exit codes, empty collection, JSON schema).

GUI walkthrough (3 steps)

Open http://localhost:8501 after diffvector gui.

Step 1 — Ingest (Chroma)

Click Connect & Ingest on tab ① Ingest.

Step 1 — Ingest (Qdrant)

Click Connect & Ingest.

Step 1 — Ingest (Pinecone)

Seed first (creates index demo if missing):

python -m utils.mock_generator --backend pinecone --collection demo --count 100

Click Connect & Ingest. Verify in the Pinecone console that index demo has vectors.

Step 2 — Simulate

Click Run Simulation on tab ② Simulate.

Step 3 — Diagnose

Click Run Diagnostics on tab ③ Diagnose to see the cluster shift map and retrieval overlap matrix.

Why are API keys in the sidebar?

The sidebar lists all supported integrations in one place. Keys are optional for local testing:

For first-time local testing: Chroma + local/offline-hash-384 + no keys.

Troubleshooting: 403 Forbidden on Simulate (Step 2)

This error means HuggingFace blocked the download of a sentence-transformers model (rate limits, no token, or network/proxy restrictions).

Fix (pick one):

1. Use the offline model (recommended for local testing)

   • Target model → local/offline-hash-384
   • No download, no API keys, works immediately

2. Use a cached HuggingFace model

   • If you previously downloaded MiniLM on this machine, select sentence-transformers/all-MiniLM-L6-v2 — DiffVector loads from cache first

3. Add a HuggingFace token

   # In .env or the sidebar "HuggingFace Token" field
   HF_TOKEN=hf_your_token_here
   

   Get a free token at huggingface.co/settings/tokens, or run:

   huggingface-cli login
   

Expected results with mock data

Low overlap (e.g. ~10%) is normal with the mock generator. It stores random 1536-d vectors as "old" embeddings, while your target model produces semantic vectors from the text. DiffVector is correctly detecting drift — not a bug.

Real production audits (same texts, ada-002 stored vectors vs text-embedding-3-small re-embed) will show more meaningful overlap numbers.

CLI local testing

# Chroma — offline model, no keys
diffvector audit \
  --source chroma --path ./chroma_db --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json

# Qdrant — requires Docker
diffvector audit \
  --source qdrant --url http://localhost:6333 --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json

# Pinecone — requires PINECONE_API_KEY in .env
python -m utils.mock_generator --backend pinecone --collection demo --count 100
diffvector audit \
  --source pinecone --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json

Test all backends at once (before publish)

pip install -e ".[dev]"

# One script: seed + audit Chroma, Qdrant, and Pinecone
chmod +x scripts/test-all-backends.sh
./scripts/test-all-backends.sh

What it does:

1. Seeds ./chroma_db and runs diffvector audit --source chroma
2. Starts Qdrant via Docker, seeds collection demo, runs audit
3. Seeds Pinecone index demo and runs audit — skipped with a warning if PINECONE_API_KEY is not in .env

Override defaults:

COLLECTION=my-index TARGET_MODEL=local/offline-hash-384 ./scripts/test-all-backends.sh

Full pre-publish gate (unit tests + CI exit-code checks + optional backends):

chmod +x scripts/test-cli.sh
./scripts/test-cli.sh

Run unit tests

pip install -e ".[dev]"
pytest tests/ -v

All unit tests run without API keys or Docker (connectors are tested against local Chroma fixtures and mocked Pinecone factory paths).

Seed all backends

# Chroma + Qdrant always; Pinecone only if PINECONE_API_KEY is set
python -m utils.mock_generator --backend all --collection demo --count 100

Restarting your local stack

# Full reset: Qdrant + seed both Chroma and Qdrant
./scripts/setup-local.sh

# Or step by step
export PATH="/Applications/Docker.app/Contents/Resources/bin:$PATH"  # macOS if needed
docker compose up -d
python -m utils.mock_generator --backend all
diffvector gui

Supported Backends

Supported Target Models

Copy .env.example to .env and fill in keys only when needed:

cp .env.example .env

Key Metrics

• Top-K Overlap — |old_top_k ∩ new_top_k| / k, bounded [0.0, 1.0]
• Drift Score — 1 − mean_overlap
• High-Risk Flag — overlap < 40% per query

Cross-dimension comparison (1536-d → 384-d) works because retrieval happens inside each space independently, then results are compared by document ID.

Screenshots

The 3-step engine: ingest a read-only slice, simulate a target embedding model, diagnose drift.

Step 3 metrics and cluster shift map (mock Chroma data → local/offline-hash-384).

CLI audit with --json for CI pipelines.

Regenerate screenshots after UI changes:

pip install -e ".[dev]"
python scripts/generate_readme_assets.py

Pre-publish checklist

Run this before making the repo public:

pip install -e ".[dev]"

# 1. All backends — seed + audit in one run (Pinecone needs PINECONE_API_KEY in .env)
chmod +x scripts/test-all-backends.sh
./scripts/test-all-backends.sh

# 2. Full CI gate — unit tests, exit codes, empty collection, JSON schema
chmod +x scripts/test-cli.sh
./scripts/test-cli.sh

# 3. Refresh README screenshots
python scripts/generate_readme_assets.py

# 4. Manual GUI smoke test (optional)
python -m utils.mock_generator --backend all   # Chroma + Qdrant (+ Pinecone if key set)
diffvector gui

What test-cli.sh verifies

What test-all-backends.sh verifies

Same seed + audit path as above, but requires Chroma and Qdrant to pass (exits on failure). Pinecone is tested when a key is present; otherwise it prints a skip warning and still exits 0 after Chroma + Qdrant pass.

Qdrant seed data (already included)

Qdrant test data is seeded by the mock generator — no manual upserts needed:

# Start Qdrant + seed Chroma and Qdrant together
./scripts/setup-local.sh

# Or Qdrant only
docker compose up -d
python -m utils.mock_generator --backend qdrant --url http://localhost:6333 --collection demo --count 300

Verify in the Qdrant dashboard: http://localhost:6333/dashboard → collection demo (~300 points, 1536-d cosine).

Manual CLI commands to spot-check

# Must pass (exit 0)
diffvector audit \
  --source chroma --path ./chroma_db --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json

# Must fail (exit 1) — proves CI gate works
diffvector audit \
  --source chroma --path ./chroma_db --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.99 --json

# Qdrant (after docker compose up -d + seed)
diffvector audit \
  --source qdrant --url http://localhost:6333 --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json

# Pinecone (after seed + PINECONE_API_KEY in .env)
diffvector audit \
  --source pinecone --collection demo \
  --target-model local/offline-hash-384 \
  --n 50 --min-overlap 0.0 --json

Project Structure

diffvector/
├── app.py                   # Streamlit GUI
├── cli.py                   # Typer CLI (diffvector audit)
├── core/
│   ├── connectors.py        # Chroma, Qdrant, Pinecone (read-only)
│   ├── embeddings.py        # OpenAI, Cohere, sentence-transformers
│   └── analytics.py         # t-SNE, Top-K overlap
├── utils/
│   └── mock_generator.py    # Seed Chroma / Qdrant / Pinecone for testing
├── docker-compose.yml       # Local Qdrant + optional GUI profile
├── Dockerfile               # ghcr.io/shunvel/diffvector image
├── scripts/
│   ├── setup-local.sh              # Start Qdrant + seed Chroma & Qdrant
│   ├── test-cli.sh                 # Pre-publish CLI verification
│   ├── test-all-backends.sh        # Chroma + Qdrant + Pinecone in one run
│   └── generate_readme_assets.py   # Regenerate docs/screenshots/
├── docs/
│   └── screenshots/                # README images (auto-generated)
└── tests/                   # TV-001 through TV-004

Security

• Read-only — DiffVector never modifies your vector database
• Local-first — Data stays in Streamlit session state or CLI stdout
• Keys stay local — API keys via .env or sidebar; never logged

Development

git clone https://github.com/shunvel/diffvector.git
cd diffvector
pip install -e ".[dev]"
pytest tests/ -v
diffvector gui

Publishing: see docs/PUBLISH.md.

License

MIT © 2026 shunvel