rag-staleness-check 0.1.0

Read-only staleness/orphan/duplicate/retrievability checks for a single pgvector, Qdrant, or Chroma index.

rag-staleness-check

Read-only staleness / orphan / duplicate / retrievability checks for a single pgvector, Qdrant, or Chroma index — the open-source half of the RAGproof "decayed RAG index" teardown (full writeup + multi-engine ledger-verified methodology).

This tool runs against your own already-indexed vector database and tells you:

• staleness — indexed chunks whose source document has since changed (needs a manifest with last_modified per document, and the engine itself storing a per-row last-modified value)
• orphans — indexed chunks whose source document no longer exists in your manifest at all
• duplicates — near-identical chunks (cosine similarity ≥ threshold, default 0.98), plus an exact-hash pass if you store a per-chunk content hash
• retrievable-after-delete — a basic probe: for ids you believe are deleted, is the vector still physically retrievable by id (storage-layer persistence), and does it still surface in top-k search results (functional leak)?

It is read-only: it never calls a write/delete/upsert method on your engine. See Read-only guarantee below for exactly how that's enforced, and its documented limits.

What this tool does NOT do

This is the single-engine, ledger-free, self-serve slice. It deliberately does not include:

• multi-engine orchestration (run it once per engine yourself)
• the GDPR Article-17 reporting pack (signed evidence, verifiable-erasure report)
• engine-specific deep checks (pgvector dead-tuple/VACUUM detail, Qdrant optimizer-threshold detail, Chroma on-disk HNSW growth)
• a ledger-based precision/recall harness (there's no ground truth here — this tool reports what it finds, not how accurate the finding is)

Those live in the private, paid RAGproof audit, which cross-validates every finding against a git-history-derived ground-truth ledger across all three engines simultaneously.

Install

pip install rag-staleness-check              # pgvector only
pip install "rag-staleness-check[qdrant]"     # + Qdrant
pip install "rag-staleness-check[chroma]"     # + Chroma
pip install "rag-staleness-check[all]"        # everything

Or, without installing:

pipx run rag-staleness-check --engine pgvector --dsn "$DSN" --source ./docs_manifest.json

Requires Python ≥ 3.10.

Usage

rag-staleness-check \
  --engine pgvector \
  --dsn "postgresql://readonly_user:pw@localhost:5432/mydb" \
  --pg-table chunks \
  --pg-doc-id-column doc_id \
  --pg-last-modified-column last_modified \
  --pg-content-hash-column content_sha256 \
  --source ./docs_manifest.json \
  --deleted-ids ./deleted_ids.json \
  --out findings.json

Prints a scorecard to stdout and writes the full result to --out (default findings.json). Every check that's missing a required input (no --source, no per-row metadata configured, no --deleted-ids) reports "skipped": true with a reason — it never silently reports a misleadingly-clean 0%.

--dsn per engine

Engine	--dsn format	Example
pgvector	Postgres DSN	postgresql://user:pw@localhost:5432/db
qdrant	Base URL	http://localhost:6333
chroma	host:port	localhost:8000

Schema mapping

A third party's table/column or payload/metadata field names are arbitrary — this tool doesn't assume anything about your schema beyond what you tell it via flags:

pgvector (--pg-*): --pg-table (required), --pg-id-column (default id), --pg-vector-column (default embedding), --pg-doc-id-column, --pg-last-modified-column, --pg-content-hash-column (all three optional — their absence is exactly what makes staleness / orphans / the duplicates check's exact-hash pass report skipped / zero-coverage).

Qdrant / Chroma (--collection required, plus payload/metadata field names): --doc-id-field (default doc_id), --last-modified-field (default last_modified), --content-hash-field (default content_sha256).

Manifest (--source)

A JSON file describing what documents should exist — the join key for staleness and orphans. There is no ledger in this tool; this manifest is the only source of truth about "what's current."

{
  "manifest_version": 1,
  "documents": [
    {
      "doc_id": "handbook/engineering/on-call.md",
      "last_modified": "2026-06-01T00:00:00Z",
      "content_sha256": "optional, doc-level, informational only"
    }
  ]
}

doc_id must match whatever value your own ingestion pipeline wrote into each indexed chunk's doc-id column/field (see schema mapping above). last_modified is only needed if you want the staleness check to run. content_sha256 here is doc-level and purely informational — it is not the input to duplicate detection (see below). See examples/docs_manifest.json.

Deleted-ids probe (--deleted-ids)

A JSON array of ids you believe are deleted:

["chunk-abc123", "chunk-def456"]

For each: checks whether the vector is still retrievable by id (storage-layer persistence) and, if so, whether it still surfaces in a top-k self-query (functional leak). This distinction matters — per EDPB Guidelines 05/2019, erasure must be verifiable and irreversible; suppressing a record from search results alone does not, by itself, satisfy that if the underlying vector is still present. findings.json's retrievable.framing field states this explicitly every run.

Read-only guarantee

This tool never calls a write/delete/upsert method on your engine. Enforced two ways:

1. Structurally, via a static test — tests/test_no_write_methods.py AST-walks every file under src/rag_staleness_check/ and fails the build if any write/delete/upsert-shaped method is called, or any SQL write statement is passed to execute()/executemany(). Not a substring grep (which would false-positive on this very README/these docstrings) — it parses the actual syntax tree.
2. At runtime, for pgvector — every connection is opened with SET default_transaction_read_only = on, then rag_staleness_check.readonly.assert_pg_read_only() re-checks SHOW default_transaction_read_only before any query runs, and refuses to proceed if it isn't 'on'. This is defense-in-depth against a connection pooler (e.g. PgBouncer) silently swallowing the session-level SET.

Honest limitation: Qdrant and Chroma have no client-exposed way to introspect "is this session/API key read-only" — that's deployment-side RBAC, not something either client library reports. For those two engines, enforcement is (1) the static test above and (2) your own deployment-side scoping — connect with a read-only-scoped API key/token if your engine supports one. There is no runtime assertion for Qdrant/Chroma equivalent to pgvector's; this README says so rather than implying parity.

For extra assurance on pgvector, connect through a locked-down role instead of relying on this tool alone:

CREATE ROLE rag_staleness_check_readonly NOSUPERUSER LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE mydb TO rag_staleness_check_readonly;
GRANT USAGE ON SCHEMA public TO rag_staleness_check_readonly;
GRANT SELECT ON chunks TO rag_staleness_check_readonly;

Telemetry

No default telemetry. Nothing is collected or sent automatically. --share-anonymous-scorecard is an explicit opt-in that prints the exact anonymized payload (aggregate counts + engine type only — never your dsn, hostnames, doc_ids, or chunk_ids) that would be shared — no backend is configured yet, so nothing is actually sent over the network. This flag is reserved for a future opt-in submission endpoint. DO_NOT_TRACK in the environment, if set, forces this off even if the flag is passed.

Separately: this tool explicitly disables chromadb's own client-side telemetry (anonymized_telemetry=False) when connecting to Chroma — the chromadb Python package has its own default-on PostHog-based telemetry independent of anything described above, and leaving it enabled would silently violate the "no default telemetry" guarantee for --engine chroma users even though this tool's own code never phones home. Known cosmetic issue: on chromadb==0.6.3, this setting is correctly applied (verified: client._system.settings.anonymized_telemetry is False), but one startup event (ClientStartEvent) still attempts to fire through a code path that doesn't consistently honor it, and then fails with capture() takes 1 positional argument but 3 were given — a TypeError raised while constructing the call, before any network request is made. You may see this line printed; it does not indicate data was sent.

Known limitations

• Duplicate detection's exact-hash pass needs a per-chunk content hash stored in your engine (--pg-content-hash-column / --content-hash-field). Without one, it degrades to cosine-ANN-only — reported via a warning field in the duplicates check, not silently.
• Chroma's snapshot_stats() intentionally reports only a live count — no on-disk HNSW-directory-size measurement. (The private RAGproof audit's own dev-environment version of this used a docker exec du trick against a known-local container; that has no equivalent against a real third-party deployment, so it isn't shipped here.)
• qdrant-client's own compatibility check may warn if your installed client's minor version differs from your server's by more than 1 (e.g. "Qdrant client version 1.18.0 is incompatible with server version 1.15.5"). This is non-fatal — the check still runs — but if it's noisy, pin qdrant-client to match your server's minor version yourself.
• This tool reports counts and pairs, not precision/recall. There is no ground truth available client-side to score itself against (that's what the private, ledger-based audit is for).
• Duplicate detection needs a retrievable vector per candidate — a chunk with no vector returned by get_vector is silently excluded from the cosine-ANN pass (there's nothing to query with), not counted as "not a duplicate."

Dependency notes

• qdrant-client and chromadb are optional extras (pip install rag-staleness-check[qdrant] / [chroma] / [all]) — a pgvector-only user doesn't need Chroma's heavier transitive dependencies.
• Neither is pinned to an exact version: this tool connects to a third party's already-running deployment, whose server version is out of its control, so hard-pinning (unlike a project that ships and controls its own Docker images) would just break users on anything else.

License

Apache-2.0. See LICENSE. Independent of any other RAGproof project's license.

Contributing

Issues and PRs welcome. Not yet implemented: a --method minhash surface-text-based duplicate detection mode (via datasketch) as an alternative/complement to the cosine-embedding-based approach above — PRs welcome.