rag-staleness-check

Read-only staleness, orphan, duplicate and retrievability checks for a single pgvector, Qdrant or Chroma index.

You point it at your own already-indexed vector database and it reports:

• staleness: indexed chunks whose source document has changed since they were indexed. This needs a manifest with last_modified per document, and the engine has to store a per-row last-modified value.
• orphans: indexed chunks whose source document is no longer in your manifest at all.
• duplicates: near-identical chunks (cosine similarity at or above a threshold, 0.98 by default), plus an exact-hash pass if you store a per-chunk content hash.
• retrievable-after-delete: a basic probe for ids you believe are deleted. It checks whether the vector can still be fetched by id (storage-layer persistence) and whether it still shows up in top-k search results (functional leak).

The tool is read-only and never calls a write, delete or upsert method on your engine. Read-only guarantee below explains how that's enforced and where the enforcement stops.

Scope

Each run covers one engine and has no ground truth to compare against. That means it doesn't do:

• multi-engine runs (run it once per engine)
• GDPR Article 17 reporting, such as signed evidence or a verifiable erasure report
• engine-specific deep checks (pgvector dead tuples and VACUUM, Qdrant optimizer thresholds, Chroma on-disk HNSW growth)
• precision/recall scoring. It reports what it finds, not how accurate those findings are.

Install

pip install rag-staleness-check              # pgvector only
pip install "rag-staleness-check[qdrant]"     # + Qdrant
pip install "rag-staleness-check[chroma]"     # + Chroma
pip install "rag-staleness-check[all]"        # everything

Or, without installing:

pipx run rag-staleness-check --engine pgvector --dsn "$DSN" --source ./docs_manifest.json

Requires Python 3.10+.

Usage

rag-staleness-check \
  --engine pgvector \
  --dsn "postgresql://readonly_user:pw@localhost:5432/mydb" \
  --pg-table chunks \
  --pg-doc-id-column doc_id \
  --pg-last-modified-column last_modified \
  --pg-content-hash-column content_sha256 \
  --source ./docs_manifest.json \
  --deleted-ids ./deleted_ids.json \
  --out findings.json

This prints a scorecard to stdout and writes the full result to --out (findings.json by default). If a check is missing an input it needs (no --source, no per-row metadata configured, no --deleted-ids), it reports "skipped": true with a reason, so you don't get a clean-looking 0% that means nothing.

--dsn per engine

Schema mapping

The tool makes no assumptions about your table, column, payload or metadata names beyond what you pass in flags.

For pgvector (--pg-*): --pg-table (required), --pg-id-column (default id), --pg-vector-column (default embedding), --pg-doc-id-column, --pg-last-modified-column and --pg-content-hash-column. The last three are optional. Leave them out and staleness, orphans, and the exact-hash pass of the duplicates check report skipped or zero coverage.

For Qdrant and Chroma, --collection is required, plus the payload or metadata field names: --doc-id-field (default doc_id), --last-modified-field (default last_modified) and --content-hash-field (default content_sha256).

Manifest (--source)

A JSON file listing the documents that should exist. It's the join key for staleness and orphans, and the tool's only source of truth for what counts as current.

{
  "manifest_version": 1,
  "documents": [
    {
      "doc_id": "handbook/engineering/on-call.md",
      "last_modified": "2026-06-01T00:00:00Z",
      "content_sha256": "optional, doc-level, informational only"
    }
  ]
}

doc_id has to match whatever your ingestion pipeline wrote into each chunk's doc-id column or field (see schema mapping above). last_modified is only needed for the staleness check. content_sha256 here is per document and informational only; duplicate detection doesn't use it (see below). There's a full example at examples/docs_manifest.json.

Deleted-ids probe (--deleted-ids)

A JSON array of ids you believe are deleted:

["chunk-abc123", "chunk-def456"]

For each id, the probe checks whether the vector can still be fetched by id (storage-layer persistence) and, if it can, whether it still comes back in a top-k self-query (functional leak). The two are worth telling apart. Under EDPB Guidelines 05/2019, erasure has to be verifiable and irreversible, so hiding a record from search results isn't enough if the vector is still stored. The retrievable.framing field in findings.json repeats this on every run.

Read-only guarantee

The tool never calls a write, delete or upsert method on your engine. Two things enforce that:

1. A static test, tests/test_no_write_methods.py, parses every file under src/rag_staleness_check/ and fails the build if it finds a call to a write/delete/upsert-style method, or a SQL write statement passed to execute()/executemany(). It walks the syntax tree instead of grepping for substrings, which would trip on this README and on docstrings.
2. For pgvector, a runtime check. Every connection opens with SET default_transaction_read_only = on, and rag_staleness_check.readonly.assert_pg_read_only() reads SHOW default_transaction_read_only before any query and refuses to continue unless it's 'on'. This covers the case where a connection pooler such as PgBouncer silently drops the session-level SET.

Qdrant and Chroma have no client-side way to ask "is this session or API key read-only?" That's handled by RBAC on the deployment, and neither client library reports it. For those two engines the static test is the only enforcement in this tool, so connect with a read-only API key or token if your deployment supports one. There is no runtime check for Qdrant or Chroma like the one for pgvector.

On pgvector you can go further and connect through a locked-down role:

CREATE ROLE rag_staleness_check_readonly NOSUPERUSER LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE mydb TO rag_staleness_check_readonly;
GRANT USAGE ON SCHEMA public TO rag_staleness_check_readonly;
GRANT SELECT ON chunks TO rag_staleness_check_readonly;

Telemetry

There is no default telemetry. Nothing is collected or sent automatically. --share-anonymous-scorecard is an explicit opt-in that prints the anonymized payload that would be shared: aggregate counts and the engine type, never your dsn, hostnames, doc_ids or chunk_ids. No backend is configured yet, so nothing goes over the network; the flag is reserved for a future opt-in endpoint. If DO_NOT_TRACK is set in the environment, it turns sharing off even when the flag is passed.

The tool also turns off chromadb's own client telemetry (anonymized_telemetry=False) when it connects to Chroma. The chromadb package ships PostHog-based telemetry that is on by default, and leaving it on would break the no-telemetry promise for --engine chroma even though this tool's own code never phones home.

One cosmetic quirk on chromadb==0.6.3: the setting does take effect (client._system.settings.anonymized_telemetry is False), but a startup event (ClientStartEvent) still goes through a code path that ignores it, and fails with capture() takes 1 positional argument but 3 were given while building the call, before any request is made. You may see that line printed. Nothing was sent.

Known limitations

• The exact-hash pass of duplicate detection needs a per-chunk content hash stored in your engine (--pg-content-hash-column / --content-hash-field). Without one, the check falls back to cosine ANN only and says so in the warning field of the duplicates result.
• Chroma's snapshot_stats() only reports a live count. Measuring the on-disk HNSW directory would need filesystem access to the Chroma host, which you won't have against someone else's deployment, so it isn't included.
• qdrant-client runs its own compatibility check and may warn when the client's minor version is more than one away from the server's (for example "Qdrant client version 1.18.0 is incompatible with server version 1.15.5"). The warning isn't fatal and the check still runs. If it gets noisy, pin qdrant-client to your server's minor version.
• The tool reports counts and pairs, not precision or recall. There's no ground truth on the client side to score itself against.
• Duplicate detection needs a vector for each candidate. A chunk whose vector get_vector doesn't return is left out of the cosine ANN pass, since there's nothing to query with. It isn't counted as "not a duplicate."

Dependency notes

• qdrant-client and chromadb are optional extras (pip install rag-staleness-check[qdrant] / [chroma] / [all]), so pgvector-only users don't pull in Chroma's heavier dependencies.
• Neither is pinned to an exact version. The tool connects to a deployment that's already running, and you control its server version, not this tool. Hard pins would break anyone on a different version.

License

Apache-2.0. See LICENSE.

Contributing

Issues and PRs are welcome. One open idea: a --method minhash mode that detects duplicates from surface text (via datasketch), as an alternative or complement to the cosine-embedding approach.