vecbee 0.1.0

A lightweight, open-source vector-database migration & infrastructure CLI (Alembic for vector databases).

Alembic for vector databases.

Migrate, sync, and re-embed vector data across stores with blue-green zero-downtime cutovers — all driven by one declarative vector.config.yaml.

---

Why vecbee?

Relational databases have had migration tooling for decades — Alembic, Flyway, Rails migrations. Vector databases have had almost nothing. When your embedding model changes, your chunking strategy improves, or you outgrow your vector store, you're left writing one-off scripts, praying nothing breaks, and taking downtime while you rebuild an index.

vecbee treats vector migrations like the versioned, reviewable, reversible operations they should be. It re-embeds and re-chunks into a shadow collection, verifies it, then does an atomic alias swap — your application never queries a half-built index, and if something's wrong you roll back with one command.

pip install vecbee
vecbee init      # scaffolds a runnable project — zero API keys, zero infra
vecbee up        # runs the full pipeline for $0 (mock embeddings + local store)

That first up runs end-to-end with no paid API calls and no external services, so you can see exactly what vecbee does before pointing it at anything real.

---

Features

• 🐝 Blue-green zero-downtime cutovers — migrations build a shadow collection and flip a stable alias atomically. Readers never see a partial index.
• ⏪ Reversible — vecbee down reverts the most recently applied migration.
• 🔀 Three migration modes — sql_chunk_embed (chunk + embed rows from any SQL source), re_embed (re-embed existing content with a new model), and pure_copy (move vectors verbatim, zero embedding calls).
• 📈 Incremental sync — an updated_at watermark means re-runs only process changed rows, not the whole table.
• 🔎 Drift detection — vecbee status reconciles applied migration history against your local migration files and reports drift before you touch production.
• 🧾 Dead-letter queue — every failed record is written to dlq/ as self-describing JSONL. Nothing is ever silently dropped.
• 🔁 Resumable & rate-limit aware — upsert-by-source-id makes runs idempotent; exponential backoff handles 429s from embedding APIs.
• 🧩 Pluggable — sources, targets, and embedding providers are swappable drivers; third-party plugins register via a standard entry-point group.
• 💸 Zero-cost default path — mock provider + local SQLite store means CI and first-run cost nothing and need no network.
• 🎛️ Declarative — one vector.config.yaml is the whole contract. No imperative glue code.

---

Quickstart

1. Install

pip install vecbee
# optional extras for real targets/providers:
pip install "vecbee[qdrant]"      # Qdrant vector target
pip install "vecbee[postgres]"    # Postgres/asyncpg source

2. Scaffold a project

vecbee init

This writes a runnable vector.config.yaml (wired to the mock provider + local store), a sample SQL source fixture, and a migrations/ directory.

3. Run the pipeline — for $0

vecbee up

up chunks and embeds the sample data into a shadow collection, then atomically swaps the docs alias to it. No API keys, no Docker, no cost.

4. Point it at your own data

Edit the source block in vector.config.yaml to your real database, swap provider.driver to a real embedding API, and re-run vecbee up.

---

How it works

                    ┌─────────────────────────────────────────────┐
   vector.config    │  vecbee up                                   │
   .yaml  ────────► │                                              │
                    │  source ──► chunk ──► embed ──► shadow write  │
                    │   (SQL)     (recursive)  (provider)   │       │
                    │                                       ▼       │
   migrations/  ──► │                              ┌── reconcile ──┐│
   0001_*.yaml      │                              │  verify count ││
                    │                              └──────┬────────┘│
                    │        readers query  ◄── alias ────┘         │
                    │        "docs"          atomic swap (cutover)  │
                    └─────────────────────────────────────────────┘
                    failed records ──► dlq/*.jsonl   (never dropped)

The alias (collection: in your config) is the stable name your application queries. A migration never mutates the live collection in place — it builds a new shadow collection and, only once written and reconciled, repoints the alias. That repoint is the cutover.

---

Commands

Command	What it does
vecbee init	Scaffold a runnable project (vector.config.yaml + sample fixtures + migrations/).
vecbee status	Reconcile applied migration history vs. local migration files; report drift.
vecbee up	Blue-green migration into a shadow collection + atomic cutover.
vecbee down	Revert the most recently applied migration.
vecbee run-ext <name>	Load and run one extension standalone (no args → list discoverable extensions).
vecbee --version	Print the installed version.

---

Configuration

vecbee init generates this annotated starter. Every block below already works end-to-end against the zero-cost defaults; edit source to migrate your own data.

source:
  driver: sql                       # read rows from any SQLAlchemy-supported database
  dsn: "sqlite+aiosqlite:///./.vecbee/sample_data/sample_sql_source.db"
  table: documents
  updated_at_column: updated_at     # enables incremental sync

target:
  driver: local                     # zero-infra SQLite store with real atomic alias-swap
  collection: docs                  # the stable alias your application queries
  path: "./.vecbee/local_target.db"

provider:
  driver: mock                      # deterministic, zero-cost, offline embeddings
  dimensions: 8

chunker:
  driver: recursive                 # langchain-text-splitters RecursiveCharacterTextSplitter
  chunk_size: 1000
  chunk_overlap: 200

state_store:
  dsn: "sqlite+aiosqlite:///./.vecbee/state.db"   # tracks migration history

migrations:
  dir: "./migrations"               # your NNNN_<slug>.yaml migration files

dlq:
  path: "./dlq"                     # failed records land here as JSONL

concurrency: 8                      # in-flight records
batch_size: 64                      # records per batch

Values support ${ENV_VAR} interpolation for secrets (see .env.example).

Drivers & providers

Kind	Driver	Notes
Source	sql	Any SQLAlchemy dialect (SQLite, Postgres via [postgres], …).
Source	vector_local	Read vectors back out of a local store (e.g. for pure_copy).
Target	local	Zero-infra SQLite store with WAL-backed atomic alias swap.
Target	qdrant	Real Qdrant target (pip install "vecbee[qdrant]").
Provider	mock	Deterministic offline embeddings — $0, CI-safe.
Provider	openai_compat	Any OpenAI-compatible embeddings endpoint (key-gated).
Chunker	recursive	RecursiveCharacterTextSplitter with configurable size/overlap.

---

Architecture

vecbee enforces one hard rule: engine/ is pure domain logic and must never import display/ or Rich. The engine emits plain dataclasses; display/ is the only place Rich is used to render them. This boundary is enforced in CI by import-linter, so the core stays testable, scriptable, and free of presentation concerns.

src/vecbee/
├── abc/         # provider / source / target / hooks interfaces (the contracts)
├── engine/      # pure orchestration: pipeline, reconciler, drift, dlq, backoff, modes
├── drivers/     # concrete sources, targets, providers, chunker
├── config/      # vector.config.yaml loading, validation, secrets
├── cli/         # Typer commands (init / status / up / down / run-ext)
├── display/     # Rich rendering — the ONLY place Rich is imported
├── plugins/     # entry-point discovery for third-party drivers
└── state/       # migration-history persistence

---

Extending vecbee

Third-party drivers register under the vecbee.plugins entry-point group. In your plugin package's pyproject.toml:

[project.entry-points."vecbee.plugins"]
my_target = "my_pkg.my_module:MyTargetDriver"

Then vecbee run-ext will discover it, and it becomes usable as a driver: in config.

---

Development

git clone https://github.com/Caoquyen1913/vecbee.git
cd vecbee
pip install -e ".[dev,qdrant]"

ruff check .        # lint
mypy                # strict type-check on engine/abc/config
lint-imports        # enforce the engine ↛ display boundary
pytest              # full test suite

The optional Qdrant integration check runs against a real dockerized Qdrant:

docker compose up -d qdrant
QDRANT_URL=http://localhost:6333 python scripts/qdrant_integration_check.py
docker compose down

---

Project status

Alpha (0.1.0). The interface is exercised by a large passing test suite and every acceptance criterion, but has no real-world usage history yet. Expect the API to stabilize toward 0.2.0. Feedback and issues are very welcome.

Contributing

Contributions are welcome! Please open an issue to discuss substantial changes first. Before submitting a PR, make sure ruff, mypy, lint-imports, and pytest all pass locally (they're gated in CI). See PUBLISHING.md for the release process.

License

Apache-2.0 © Kawin Tran (@Caoquyen1913) and AI Outsourcing Studio.