vecbee 0.1.1

A lightweight, open-source vector-database migration & infrastructure CLI (Alembic for vector databases).

Alembic for vector databases.

Migrate, sync, and re-embed vector data across stores with blue-green zero-downtime cutovers — all driven by one declarative vector.config.yaml.

---

vecbee is an open-source command-line tool for vector-database migrations — it re-embeds, re-chunks, and moves embedding data between vector stores (Qdrant, a local SQLite store, and more) with blue-green, zero-downtime cutovers, driven by a single declarative vector.config.yaml. Think Alembic or Flyway, but for vector databases and RAG pipelines.

Why vecbee?

Relational databases have had migration tooling for decades — Alembic, Flyway, Rails migrations. Vector databases have had almost nothing. When your embedding model changes, your chunking strategy improves, or you outgrow your vector store, you're left writing one-off scripts, praying nothing breaks, and taking downtime while you rebuild an index.

vecbee treats vector migrations like the versioned, reviewable, reversible operations they should be. It re-embeds and re-chunks into a shadow collection, verifies it, then does an atomic alias swap — your application never queries a half-built index, and if something's wrong you roll back with one command.

pip install vecbee
vecbee init      # scaffolds a runnable project — zero API keys, zero infra
vecbee up        # runs the full pipeline for $0 (mock embeddings + local store)

That first up runs end-to-end with no paid API calls and no external services, so you can see exactly what vecbee does before pointing it at anything real.

---

Features

• 🐝 Blue-green zero-downtime cutovers — migrations build a shadow collection and flip a stable alias atomically. Readers never see a partial index.
• ⏪ Reversible — vecbee down reverts the most recently applied migration.
• 🔀 Three migration modes — sql_chunk_embed (chunk + embed rows from any SQL source), re_embed (re-embed existing content with a new model), and pure_copy (move vectors verbatim, zero embedding calls).
• 📈 Incremental sync — an updated_at watermark means re-runs only process changed rows, not the whole table.
• 🔎 Drift detection — vecbee status reconciles applied migration history against your local migration files and reports drift before you touch production.
• 🧾 Dead-letter queue — every failed record is written to dlq/ as self-describing JSONL. Nothing is ever silently dropped.
• 🔁 Resumable & rate-limit aware — upsert-by-source-id makes runs idempotent; exponential backoff handles 429s from embedding APIs.
• 🧩 Pluggable — sources, targets, and embedding providers are swappable drivers; third-party plugins register via a standard entry-point group.
• 💸 Zero-cost default path — mock provider + local SQLite store means CI and first-run cost nothing and need no network.
• 🎛️ Declarative — one vector.config.yaml is the whole contract. No imperative glue code.

---

Quickstart

1. Install

pip install vecbee
# optional extras for real targets/providers:
pip install "vecbee[qdrant]"      # Qdrant vector target
pip install "vecbee[postgres]"    # Postgres/asyncpg source

2. Scaffold a project

vecbee init

This writes a runnable vector.config.yaml (wired to the mock provider + local store), a sample SQL source fixture, and a migrations/ directory.

3. Run the pipeline — for $0

vecbee up

up chunks and embeds the sample data into a shadow collection, then atomically swaps the docs alias to it. No API keys, no Docker, no cost.

4. Point it at your own data

Edit the source block in vector.config.yaml to your real database, swap provider.driver to a real embedding API, and re-run vecbee up.

---

How it works

The alias (collection: in your config) is the stable name your application queries. A migration never mutates the live collection in place — it builds a new shadow collection and, only once written and reconciled, repoints the alias. That repoint is the cutover.

---

Commands

Command	What it does
vecbee init	Scaffold a runnable project (vector.config.yaml + sample fixtures + migrations/).
vecbee status	Reconcile applied migration history vs. local migration files; report drift.
vecbee up	Blue-green migration into a shadow collection + atomic cutover.
vecbee down	Revert the most recently applied migration.
vecbee run-ext <name>	Load and run one extension standalone (no args → list discoverable extensions).
vecbee --version	Print the installed version.

---

Configuration

vecbee init generates this annotated starter. Every block below already works end-to-end against the zero-cost defaults; edit source to migrate your own data.

source:
  driver: sql                       # read rows from any SQLAlchemy-supported database
  dsn: "sqlite+aiosqlite:///./.vecbee/sample_data/sample_sql_source.db"
  table: documents
  updated_at_column: updated_at     # enables incremental sync

target:
  driver: local                     # zero-infra SQLite store with real atomic alias-swap
  collection: docs                  # the stable alias your application queries
  path: "./.vecbee/local_target.db"

provider:
  driver: mock                      # deterministic, zero-cost, offline embeddings
  dimensions: 8

chunker:
  driver: recursive                 # langchain-text-splitters RecursiveCharacterTextSplitter
  chunk_size: 1000
  chunk_overlap: 200

state_store:
  dsn: "sqlite+aiosqlite:///./.vecbee/state.db"   # tracks migration history

migrations:
  dir: "./migrations"               # your NNNN_<slug>.yaml migration files

dlq:
  path: "./dlq"                     # failed records land here as JSONL

concurrency: 8                      # in-flight records
batch_size: 64                      # records per batch

Values support ${ENV_VAR} interpolation for secrets (see .env.example).

Drivers & providers

Kind	Driver	Notes
Source	sql	Any SQLAlchemy dialect (SQLite, Postgres via [postgres], …).
Source	vector_local	Read vectors back out of a local store (e.g. for pure_copy).
Target	local	Zero-infra SQLite store with WAL-backed atomic alias swap.
Target	qdrant	Real Qdrant target (pip install "vecbee[qdrant]").
Provider	mock	Deterministic offline embeddings — $0, CI-safe.
Provider	openai_compat	Any OpenAI-compatible embeddings endpoint (key-gated).
Chunker	recursive	RecursiveCharacterTextSplitter with configurable size/overlap.

---

Architecture

vecbee enforces one hard rule: engine/ is pure domain logic and must never import display/ or Rich. The engine emits plain dataclasses; display/ is the only place Rich is used to render them. This boundary is enforced in CI by import-linter, so the core stays testable, scriptable, and free of presentation concerns.

src/vecbee/
├── abc/         # provider / source / target / hooks interfaces (the contracts)
├── engine/      # pure orchestration: pipeline, reconciler, drift, dlq, backoff, modes
├── drivers/     # concrete sources, targets, providers, chunker
├── config/      # vector.config.yaml loading, validation, secrets
├── cli/         # Typer commands (init / status / up / down / run-ext)
├── display/     # Rich rendering — the ONLY place Rich is imported
├── plugins/     # entry-point discovery for third-party drivers
└── state/       # migration-history persistence

---

Extending vecbee

Third-party drivers register under the vecbee.plugins entry-point group. In your plugin package's pyproject.toml:

[project.entry-points."vecbee.plugins"]
my_target = "my_pkg.my_module:MyTargetDriver"

Then vecbee run-ext will discover it, and it becomes usable as a driver: in config.

---

Development

git clone https://github.com/Caoquyen1913/vecbee.git
cd vecbee
pip install -e ".[dev,qdrant]"

ruff check .        # lint
mypy                # strict type-check on engine/abc/config
lint-imports        # enforce the engine ↛ display boundary
pytest              # full test suite

The optional Qdrant integration check runs against a real dockerized Qdrant:

docker compose up -d qdrant
QDRANT_URL=http://localhost:6333 python scripts/qdrant_integration_check.py
docker compose down

---

vecbee vs. a hand-rolled migration script

	Hand-rolled script	vecbee
Zero-downtime cutover	Manual, error-prone	✅ Atomic alias swap, built in
Rollback	Rewrite/rerun by hand	✅ vecbee down
Failed records	Often silently lost	✅ Dead-letter queue (JSONL)
Resume after crash	Usually restart from zero	✅ Idempotent upsert-by-source-id
Rate-limit handling	Ad-hoc sleep()	✅ Exponential backoff
Drift / history	None	✅ vecbee status reconciles history
Config	Scattered in code	✅ One declarative vector.config.yaml
Cost to try	—	✅ $0 mock + local default path

FAQ

What is vecbee? vecbee is an open-source Python CLI for migrating vector-database data — re-embedding, re-chunking, and moving embeddings between vector stores with blue-green, zero-downtime cutovers. It's often described as "Alembic for vector databases."

Is vecbee like Alembic or Flyway, but for vector databases? Yes. Alembic and Flyway version and migrate relational schemas; vecbee versions and migrates vector data — the embeddings, chunks, and collections that power RAG and semantic search.

Which vector databases does vecbee support? Today: Qdrant and a zero-infrastructure local SQLite-backed vector store (with real atomic alias-swap semantics). More targets are on the roadmap, and the driver/plugin system lets you add your own (Pinecone, Weaviate, pgvector, …).

How does vecbee achieve zero-downtime migrations? It uses a blue-green cutover: the migration writes into a new shadow collection, verifies it by reconciling record counts, then atomically repoints a stable alias. Readers always query the alias, so they never observe a partial or half-rebuilt index.

When should I use vecbee instead of a custom re-embedding script? When you care about not taking downtime, being able to roll back, not losing failed records, resuming after a crash, and having a reviewable migration history — vecbee gives you all of that out of the box instead of reimplementing it per script.

Does vecbee need an API key or a paid service to try? No. The default path uses a mock embedding provider and a local SQLite store, so vecbee init && vecbee up runs end-to-end with zero API keys, zero network, and zero cost.

What embedding providers does vecbee work with? A deterministic mock provider (for CI and first runs) and any OpenAI-compatible embeddings endpoint via the openai_compat driver. More providers can be added as plugins.

Is vecbee free and open source? Yes — Apache-2.0 licensed, installable from PyPI with pip install vecbee.

Project status

Alpha (0.1.x). The interface is exercised by a large passing test suite and every acceptance criterion, but has no real-world usage history yet. Expect the API to stabilize toward 0.2.0. Feedback and issues are very welcome.

Contributing

Contributions are welcome! Please open an issue to discuss substantial changes first. Before submitting a PR, make sure ruff, mypy, lint-imports, and pytest all pass locally (they're gated in CI). See PUBLISHING.md for the release process.

License

Apache-2.0 © Kawin Tran (@Caoquyen1913) and AI Outsourcing Studio.