embedsync

Incremental synchronization between source documents and vector indexes — detect changes, re-embed only deltas, and delete stale chunks.

Status: v0.5 — hash + Ollama embedders, paragraph chunks, JSONL destination, chunk-level re-embed, --full-reindex.

60-second try

docker compose run --rm plan  # plan sync for examples/docs
docker compose run --rm test  # pytest

Why this vs alternatives

Problem

RAG indexes rot when documents change. Full re-embeds are expensive and miss deletes. Every team rebuilds change detection from scratch.

Key features (v0.4)

• Content-hash change detection per document
• Sync plan: add / update / delete actions
• Hash embedder for offline/CI (--embedder hash)
• Ollama embedder (--embedder ollama or ollama:nomic-embed-text)
• JSONL or in-memory destination
• Unchanged docs/chunks skip re-embedding on the next run

Architecture

embedsync run ./docs
    ├── LocalFileSource
    ├── StateStore (SQLite)
    ├── plan_sync() → diff
    └── Destination (Memory → pgvector next)

Installation

pip install embedsync
pip install -e ".[dev]"

Usage

embedsync health
embedsync plan examples/docs --state-db /tmp/embedsync-demo.db
embedsync run examples/docs --dry-run --state-db /tmp/embedsync-demo.db
embedsync run examples/docs --embedder hash --destination memory --state-db /tmp/embedsync-demo.db
embedsync run examples/docs --full-reindex --embedder hash --destination jsonl:/tmp/index.jsonl
embedsync run examples/docs --embedder hash --destination jsonl:/tmp/index.jsonl
# Requires a running Ollama with an embedding model:
embedsync run examples/docs --embedder ollama --destination jsonl:/tmp/index.jsonl
embedsync run examples/docs --embedder ollama:nomic-embed-text --destination memory

Docker

docker compose run --rm test
docker compose run --rm plan

Configuration

Ollama uses OLLAMA_HOST when set (otherwise the embedder default host).

Roadmap

• Pluggable embedder protocol + hash backend
• JSONL destination (local stand-in)
• Chunk-level stable IDs across edits
• Ollama embedder (--embedder ollama)
• pgvector and Qdrant destinations
• Notion and sitemap sources

License

MIT

Known limitations (v0.4)

• Hash embeddings are not semantic — use --embedder ollama for local semantic vectors
• JSONL is not a vector DB
• Local markdown files only
• Re-runs reuse .embedsync/state.db; pass --state-db for an isolated plan
• Ollama must already be running and have the embedding model pulled