embedsync

Incremental synchronization between source documents and vector indexes — detect changes, re-embed only deltas, and delete stale chunks.

Status: v0.6 — hash + Ollama embedders, paragraph chunks, JSONL + pgvector destinations, chunk-level re-embed, --full-reindex.

60-second try

pip install embedsync
embedsync plan examples/docs --state-db /tmp/embedsync-demo.db
# or with Docker:
docker compose run --rm plan

Why this vs alternatives

Problem

RAG indexes rot when documents change. Full re-embeds are expensive and miss deletes. Every team rebuilds change detection from scratch.

Key features (v0.6)

• Content-hash change detection per document
• Sync plan: add / update / delete actions
• --full-reindex to force re-embed of all current docs
• Hash embedder for offline/CI (--embedder hash)
• Ollama embedder (--embedder ollama or ollama:nomic-embed-text)
• JSONL, in-memory, or pgvector destination
• Unchanged docs/chunks skip re-embedding on the next run

Architecture

embedsync run ./docs
    ├── LocalFileSource
    ├── StateStore (SQLite)
    ├── plan_sync() → diff
    └── Destination (Memory / JSONL / pgvector)

Installation

pip install embedsync
pip install 'embedsync[pg]'   # optional: pgvector destination
pip install -e ".[dev]"

Usage

embedsync health
embedsync plan examples/docs --state-db /tmp/embedsync-demo.db
embedsync run examples/docs --dry-run --state-db /tmp/embedsync-demo.db
embedsync run examples/docs --embedder hash --destination memory --state-db /tmp/embedsync-demo.db
embedsync run examples/docs --full-reindex --embedder hash --destination jsonl:/tmp/index.jsonl
embedsync run examples/docs --embedder hash --destination jsonl:/tmp/index.jsonl
# Requires Postgres with pgvector + pip install 'embedsync[pg]':
embedsync run examples/docs --embedder hash --destination pgvector:postgresql://user:pass@localhost/db
# Requires a running Ollama with an embedding model:
embedsync run examples/docs --embedder ollama --destination jsonl:/tmp/index.jsonl
embedsync run examples/docs --embedder ollama:nomic-embed-text --destination memory

Docker

docker compose run --rm test
docker compose run --rm plan

Configuration

Ollama uses OLLAMA_HOST when set (otherwise the embedder default host).

Roadmap

• Pluggable embedder protocol + hash backend
• JSONL destination (local stand-in)
• Chunk-level stable IDs across edits
• Ollama embedder (--embedder ollama)
• pgvector destination
• Qdrant destination
• Notion and sitemap sources

License

MIT

Known limitations (v0.6)

• Hash embeddings are not semantic — use --embedder ollama for local semantic vectors
• JSONL is not a vector DB; use --destination pgvector:... for Postgres
• Local markdown files only
• Re-runs reuse .embedsync/state.db; pass --state-db for an isolated plan
• Ollama must already be running and have the embedding model pulled
• pgvector destination needs pip install 'embedsync[pg]' and the vector extension