ragfresh

Detect stale, drifted, and ghost documents in RAG vector indexes.

Embeddings get created once. Source documents keep changing. Nothing tells you your index has drifted from reality until a user gets an answer grounded in a document that was edited — or deleted — weeks ago. ragfresh closes that gap with a lightweight sidecar ledger and a CI-gateable check.

Staleness classes

• unchanged — source content hash matches what was indexed
• changed, not reindexed — the source doc was edited since it was indexed
• deleted, still indexed — the source doc is gone but the vector store still serves it (ghost citation risk — the worst class)
• not yet indexed — present in the source, absent from the ledger/store

Quickstart

pip install ragfresh

ragfresh init ./docs --db ragfresh.db
# ... time passes, docs change ...
ragfresh check ./docs --db ragfresh.db

Exits non-zero when changed, not reindexed or deleted, still indexed is non-empty, so it's usable as a CI gate.

check only detects staleness. To resolve it -- re-embed changed/missing docs and remove ghost entries from both the store and the ledger -- run:

ragfresh reindex ./docs --db ragfresh.db --chroma-collection my-index

reindex uses a local embedding model (no API calls) and requires a writable store -- Chroma or pgvector; the default ledger-only store is read-only. Preview what it would do without writing anything:

ragfresh reindex ./docs --db ragfresh.db --chroma-collection my-index --dry-run

Sources & stores

• Sources: filesystem, git (ragfresh.sources.GitSource)
• Stores:
  • Chroma — pip install ragfresh[chroma], then --chroma-collection NAME [--chroma-path DIR]
  • pgvector (any Postgres table with a doc_id column and a vector-typed embedding column) — pip install ragfresh[pgvector], then --pg-dsn DSN [--pg-table documents] [--pg-id-column id] [--pg-embedding-column embedding] [--pg-document-column TEXT_COL] (the embedding/document columns are only needed for reindex, not check)
  • Neither flag given → the ledger itself stands in as the index (read-only)

Semantic drift scoring

Hash-based staleness treats any byte-level change as "changed" — a typo fix and a rewritten paragraph look identical. --score-drift classifies each changed_not_reindexed doc as cosmetic or meaningful using cosine distance between embeddings of the old and new text:

ragfresh init ./docs --db ragfresh.db          # snapshots content by default
ragfresh check ./docs --db ragfresh.db --score-drift
#   [CHANGED] report.md (meaningful, score=0.812)

Requires ragfresh[chroma] (reuses its bundled local ONNX MiniLM model — downloaded once, then fully offline, no API calls) and content snapshots from init (pass --no-store-content at init time to opt out and disable this later). Tune sensitivity with --drift-threshold (default 0.15).

By default drift scoring uses chromadb's bundled MiniLM model. If your production embeddings diverge enough from that model that drift scores wouldn't track them well, point --embedding-model (on check and reindex) at any sentence-transformers model name instead (requires the sentence-transformers package).

Chunked indexes

Most production RAG pipelines chunk a doc into several vectors (ids like report.md#0, report.md#1, ...), not one vector per doc. By default ragfresh assumes indexed ids equal doc_ids -- pass --chunk-separator on check to fold chunk ids back to their source doc:

ragfresh check ./docs --db ragfresh.db --chroma-collection my-index --chunk-separator "#"

A doc counts as indexed if any of its chunks are present. ragfresh doesn't chunk on your behalf -- it only needs to know how to reverse whatever chunking scheme you already use, via a separator (CLI) or a chunk_id_to_doc_id callable (library API, ragfresh.check.check). Detection-only: reindex doesn't support chunked stores yet -- it writes one vector per doc_id, so re-embedding into a chunked index would require ragfresh to own the chunking strategy, which it deliberately doesn't.

Development

pip install -e ".[dev]"
pytest
ruff check .
mypy

Status

v0.1.0. Filesystem + git sources; SQLite ledger; Chroma + pgvector stores (both readable and writable, for check and reindex); CLI with drift scoring (pluggable embedding model), chunk-aware detection, and reindex (flat indexes only, with --dry-run); staleness-injection benchmark (benchmarks/staleness_injection/); CI across 3 OSes × 3 Python versions plus a lint/type-check job (ruff + mypy). 44 tests passing.

Known gap: reindex doesn't support chunked indexes (writes one vector per doc_id; check handles chunking, reindex deliberately doesn't own a chunking strategy).

License

MIT