compass-kb-validation 2.2.0

Knowledge-base readiness validation for RAG & AI agents — config-driven, deterministic pipeline integrity checks.

COMPASS — the quality gate for your RAG knowledge base

Point COMPASS at your RAG knowledge base. It runs 73 deterministic checks across storage, knowledge graph, vectors and retrieval, and fails your CI build when the KB is broken. No LLM, no test-writing — the checks ship with it.

Think of it as a linter for your RAG data plane: a catalogue of built-in checks with HARD/WARN severity tiers, tuned in YAML, gating CI by exit code. It answers "is the knowledge base itself sound?" — not "are the answers good?" (that's the eval layer's job; COMPASS is the gate that runs first).

Your KB is broken and you don't know yet

Chunks that never got embedded. An embedding model swapped without a re-index. A document deleted at the source but still answering questions. Citations pointing at the wrong file. Storage, graph and vector store quietly disagreeing about what exists. All of it is invisible from the outside — retrieval still returns something, answers still look plausible, and your eval scores drift for reasons nobody can trace.

COMPASS finds these deterministically, before your agents serve them.

30 seconds to see it work

pip install compass-kb-validation
compass demo                                   # sample report, no backend needed
compass check redis://localhost:6379/my_index  # your real index: profile -> validate -> report

compass check needs no config: it profiles the index, fills in the shape it detects, runs the checks, and opens an HTML report. See QUICKSTART.md for the five-step path to a gating CI check.

Does it actually catch things?

Yes — and you can verify that claim yourself in one command, no services required:

python benchmark/broken_kb_benchmark.py
#   Detection rate:      16/16  (100%)
#   False-positive rate: 0/6    (0%)

The broken-KB benchmark injects 16 realistic faults (zero chunks, wrong embedding dimension, degenerate vectors, cross-document contamination, orphaned sources, retrieval drift, …) into fixtures, runs the real validators against them, and scores detection vs false positives. It also runs 6 healthy fixtures to prove COMPASS doesn't cry wolf. It's part of the test suite, so the number can't rot.

What COMPASS Does

COMPASS validates a document's journey through your ingestion pipeline and verifies the resulting knowledge base is agent-ready. The pipeline is declarative — you choose which steps run, in what order, in YAML — and ships with these built-in steps:

API Upload → Object Storage → Knowledge Graph → Vector Store → Retrieval → Cross-Stage Consistency

Enable only the steps your setup has (no object store? no graph? drop them), reorder them, or register your own. For each enabled step it deterministically checks, for example:

• Ingestion — the document was accepted and assigned an id (GraphQL or REST ingestion APIs, config-driven)
• Object storage — it persisted with the right content type, size, and integrity, and (opt-in) the source document is healthy across PDF/DOCX/PPTX/HTML (S3, MinIO, R2, Azure Blob, GCS…)
• Knowledge graph — metadata is present, correct, and agrees with the other stores: required predicates, identity uniqueness, storage-reference integrity, RDF typing, ontology-namespace consistency, dangling links, orphan nodes, version supersession (GraphDB, Neptune, Neo4j, Stardog, Blazegraph…) — 19 checks, HARD / WARN tiered, every predicate and query config-driven so they fit any ontology
• Vector store — chunks exist with valid embeddings, correct dimensions/metric, sane ordering/overlap, structure coherence, retrievable via KNN, no cross-document contamination, no near-duplicate bloat (Redis, Qdrant, pgvector, Chroma, Pinecone, Weaviate) — 30 toggleable checks, each on a HARD / WARN severity tier (only HARD gates the run)
• Retrieval — the document's own content is actually retrievable (recall@K, MRR); opt-in embedding-drift, provenance, filtered-recall, negative-retrieval, keyword-completeness
• Cross-stage consistency — do object storage, the graph, and the vector index agree about each document (eTag, metadata, ACL propagation)?

Two modes: live validation as documents are ingested, and post-hoc discovery of a corpus that's already loaded.

Scope note: COMPASS validates pipeline & data integrity. It does not judge answer quality — that's the evaluation layer's job. COMPASS is the gate that runs first.

Key Features

✅ Declarative pipeline — choose/reorder/extend steps in YAML; no fixed shape
✅ Config-Driven — zero code changes per project; all customization in YAML
✅ Pluggable backends — storage / vector store / knowledge graph via a provider registry (one file to add a vendor). Redis, Qdrant, pgvector, Chroma, Pinecone and Weaviate all support metadata pre-filters, so a probe reproduces production's filtered retrieval on any of them (SUPPORT.md)
✅ Pluggable steps — register custom validators in a step registry; the report adapts automatically
✅ Deterministic & cheap — structural/byte/schema checks, no LLM, CI-gateable
✅ Query & analyze the KB — ask it questions, expose it to agents over MCP, and preview embedding-model migrations / map retrieval coverage — all deterministic, read-only (see Beyond validation)
✅ Interactive reporting — self-contained HTML dashboard + JSON + JUnit XML
✅ Trend Tracking — SQLite historical performance analysis
✅ Notifications — Slack/Teams webhooks for alerts
✅ Parallel Execution — process many documents concurrently
✅ Graceful Degradation — non-critical failures don't crash the pipeline

Install

Core deps only; add extras per backend (Redis is built in):

pip install compass-kb-validation
pip install "compass-kb-validation[qdrant,pgvector,chroma]"
#   vector stores: qdrant | pgvector | chroma | pinecone | weaviate
#   also: pdf (PDF source checks) | content (HTML extraction) | azure | gcs | dashboard

Project path (saved config)

Then use the unified compass CLI - one verb, subcommands:

compass init my-kb          # scaffold config/projects/my-kb.yaml + .env.example + a .compass marker
cp .env.example .env        # fill in your backend connection
#   edit config/projects/my-kb.yaml  (search_index, document_id_field)
compass doctor --project my-kb     # verify connectivity + config (per stage)
compass discover --project my-kb   # validate the already-populated KB, emit a report

compass init drops a .compass marker, so the CLI works from any subdirectory of the project (no env var to set). Other subcommands:

compass validate --project my-kb --output-format all   # live ingestion validation
compass profile  --project my-kb                        # read a store's shape, suggest config
compass list-checks                                     # check catalogue + severity tiers (no project needed)
compass <cmd> --help                                    # per-command options

The standalone compass-discover / compass-validate / compass-profile commands keep working. Live validation also reads a test_data/<name>.yaml (scaffold it with compass-validate --project my-kb --init-test-data).

Beyond validation — inspect, migrate, map coverage

These read-only, deterministic (no-LLM) tools help you verify, diagnose, and safely change a knowledge base — they don't turn COMPASS into a retrieval app or an agent-serving platform. compass ask is a retrieval probe (see what the KB returns for a query, no generated answer); compass mcp is an inspection surface (read-only KB tools an agent can call, not a serving layer). Everything here stays on COMPASS's side of the line and never affects a validation pass/fail. They read an embedding: block in the project YAML (provider openai / azure_openai / bedrock, model_id, dimensions) so a query is embedded the way ingestion was.

# Ask the KB a question - returns the ranked chunks it would retrieve (no generated answer)
compass ask --project my-kb "what is the refund policy?"

# Match production retrieval with metadata pre-filters (hybrid search; commas AND, | ORs a field)
compass ask --project my-kb --filter "category=report,region=US|EU" "summarize the latest report"

# Expose the KB to agents (Copilot/Claude) as MCP tools over stdio
compass mcp --project my-kb --print-config   # prints a .vscode/mcp.json snippet
compass mcp --project my-kb                   # run the server (query_kb, list_documents, ...)

# Preview an embedding-model switch BEFORE you cut over (self-bootstraps, no questions needed)
compass migrate --project my-kb --to openai/text-embedding-3-large

# Map retrieval coverage / find dead zones a question set never reaches (feed real query logs)
compass coverage --project my-kb --questions query_logs.jsonl

# Prove retrieval matches production: compare vs a ground-truth file (precision/recall/Jaccard)
compass compare --project my-kb --ground-truth production_ground_truth.json
# Mixed ground-truth file? Non-retrieval cases (no chunks) are auto-skipped; filter by variant:
compass compare --project my-kb --ground-truth gt.json --variant answerable

# Hand retrieval to any eval framework: emit generic RAG eval test cases
compass ask --project my-kb --batch questions.jsonl --format eval --output eval_input.jsonl

# Corpus health (deterministic, no LLM): score chunk quality / find stale + orphan sources
compass quality   --project my-kb
compass freshness --project my-kb --stale-after 30

migrate and coverage also render a self-contained HTML report (same look as the validation dashboard) with --html [path] --open — a coverage heatmap, a migration stability distribution, verdicts, and the most-changed / dead-zone tables.

Same boundary as the gate: these never judge answer quality (that's the eval layer) and never put an LLM in a pass/fail decision. migrate and coverage share one deterministic retrieval-analysis engine (sample → in-memory KNN → neighbor/reachability analysis).

Catch regressions over time (baselines + drift)

Snapshot a known-good run, then diff later runs against it — so a scheduled check is a gate, not just a report:

compass discover --project my-kb --save-baseline          # freeze a green run -> baselines/my-kb.json
# ...later (cron / CI)...
compass discover --project my-kb --against baselines/my-kb.json --fail-on-regression

Drift flags what a single run can't: dropped documents, docs that passed before and fail now (and which checks newly fail), index drift (embedding dimension / distance metric changed under you), and readiness / pass-rate decline. --fail-on-regression makes the run exit non-zero, so cron or CI alerts on degradation. (compass check takes the same --save-baseline / --against flags.)

> From source instead of PyPI: `git clone … && cd compass && pip install -e ".[all]"`.

### Running scenarios without editing YAML

`compass-discover` takes overrides that win over config + profiling, so you can try
different shapes and policies from the command line:

```bash
# Point at a different index/identity, faster, and relax the readiness bar
compass-discover --project my-kb --parallel --workers 8 \
  --id-field agent_id --expected-dim 1024 \
  --readiness-threshold 0.99 \
  --disable-check chunk_count near_duplicate_embeddings \
  --severity duplicate_detection=hard \
  --unreachable-as skipped

Severity model: every vector check is HARD (gates the run) or WARN (advisory). --severity name=hard|warn (or redis.check_severity in YAML) promotes/demotes any check; --list-checks shows the defaults.

Releases are published via PUBLISHING.md.

Reports are written to reports/ (HTML dashboard + JSON + JUnit XML). Add --live to open a real-time progress dashboard, --trends to record history, --parallel for concurrency.

Try it locally (no cloud needed)

A self-contained Docker stack runs the full pipeline against MinIO + Redis Stack + a mock ingestion API — see integration/README.md.

Project Structure

compass/
├── run_validation.py              # Entry point — live ingestion validation
├── discover_and_validate.py       # Entry point — post-hoc discovery
├── ingestion_validation/          # Main package
│   ├── models/                    # StepResult / PipelineResult (stdlib dataclasses)
│   ├── config/                    # Layered YAML config + typed Settings
│   ├── utils/                     # Run-id, logging, polling w/ backoff
│   ├── providers/                 # Pluggable backends via a registry:
│   │   ├── storage.py             #   object storage (S3-family…)
│   │   ├── vectorstore.py         #   vector store (Redis, Qdrant, pgvector, Chroma…)
│   │   ├── graph.py               #   knowledge graph (SPARQL, Neptune…)
│   │   ├── embedding.py           #   text→vector for query (OpenAI/Azure/Bedrock)
│   │   └── source.py              #   document discovery (storage|index|manifest)
│   ├── validators/                # BaseValidator + step registry + the steps
│   │   └── registry.py            #   declarative pipeline (StepSpec registry)
│   ├── pipeline/                  # Orchestrator (context propagation, halting)
│   ├── report/                    # Self-contained HTML dashboard + JSON/JUnit
│   ├── cli.py                     # Unified `compass` CLI (all subcommands)
│   ├── ask.py                     # `compass ask` — deterministic NL retrieval
│   ├── mcp_server.py              # `compass mcp` — agent-callable KB tools (stdio)
│   ├── retrieval_analysis.py      # engine for `migrate`/`coverage`/`compare` (read-only)
│   ├── eval_interop.py            # vendor-neutral eval export + ground-truth reader
│   ├── corpus.py                  # Corpus-level KB readiness analysis
│   ├── live_dashboard.py          # FastAPI + SSE real-time dashboard
│   ├── notifications.py           # Slack / Teams webhooks
│   └── trend_tracker.py           # SQLite historical trends
├── config/{base.yaml, projects/_template.yaml}
├── test_data/_template.yaml       # Documents to validate (live mode)
├── integration/                   # Local Docker integration stack
├── tests/                         # Unit tests (backends faked — no live services)
└── reports/                       # Generated reports (gitignored)

Documentation

• QUICKSTART.md — pip install → a gating CI check in five steps
• ARCHITECTURE.md — how COMPASS is built: layers, data flow, the registries, the two run modes, extension points
• CONTRACT.md — the Knowledge-Base Contract: express your KB's guarantees as one declarative artifact, with each clause mapped to the command/flag that enforces it in CI
• CASE_STUDIES.md — the reproducible benchmark story in full, plus templates for real-outcome case studies (missing docs, filter regressions, migration drift, orphan chunks)
• benchmark/ — the broken-KB benchmark: injects realistic faults and scores detection rate (currently 16/16) vs false-positive rate (0/6)
• SUPPORT.md — backend support matrix (which backends are verified vs adapter-available)
• COMPATIBILITY.md — stability & versioning policy: what's stable, what's evolving, deprecation & pinning
• EXTENDING-BACKENDS.md — add a new backend (storage/vector/graph/source) or a new pipeline step
• integration/README.md — run the full pipeline locally on Docker
• CONTRIBUTING.md · SECURITY.md

Configuration

All behaviour lives in YAML; credentials are referenced by environment-variable name and resolved at runtime (never stored in config). config/base.yaml holds shared defaults; each config/projects/<name>.yaml overlays only what differs and selects a dev/staging/prod environment block.

# config/projects/my-kb.yaml
display_name: "My Knowledge Base"

# Declarative pipeline — choose which steps run, in what order (registry keys).
# Omit to use the full default pipeline.
pipeline:
  steps: [api_upload, s3_storage, redis_chunks, retrieval_quality]
  critical_steps: ["API Upload", "S3 Storage"]   # failure here halts the run

# Where post-hoc discovery finds documents: storage | vectorstore | manifest
discovery:
  provider: vectorstore           # enumerate the KB straight from the index

environments:
  dev:
    api:
      base_url_env: API_BASE_URL          # env-var NAME, not the value
      graphql_mutation: |
        mutation ($file: Upload!, $metadata: JSON) { uploadDocument(file:$file, metadata:$metadata){ status document_id } }
    s3:
      provider: s3                        # s3 | minio | r2 | azure_blob | gcs …
      bucket_name: my-documents
      prefix: knowledge-base/
    redis:
      provider: redis                     # redis | qdrant | pinecone | pgvector …
      search_index: my_embeddings
      expected_embedding_dim: 1536
      expected_distance_metric: COSINE

Advanced Usage

Add a custom pipeline step

Steps are declarative — register a validator and reference it in pipeline.steps:

from ingestion_validation.validators.base import BaseValidator
from ingestion_validation.validators.registry import register_step, StepSpec
from ingestion_validation.models import StepResult, StepStatus

class PiiRedactionValidator(BaseValidator):
    step_name = "PII Redaction"
    def __init__(self, config): self.config = config
    def validate(self, context: dict) -> StepResult:
        return StepResult(self.step_name, StepStatus.PASSED, "no PII leaked")

register_step(StepSpec(
    key="pii_redaction", name="PII Redaction",
    factory=lambda settings, shared: PiiRedactionValidator(settings.s3),
    abbr="PII", order=25,
))
# then:  pipeline.steps: [api_upload, pii_redaction, redis_chunks]

The orchestrator, live dashboard, and HTML report adapt automatically. See EXTENDING-BACKENDS.md for adding backends too.

CI/CD integration — the readiness gate

COMPASS is built to run before your agents or eval consume the knowledge base: wire it into the pipeline that publishes the KB and fail the build when the KB is not retrieval-ready. Every entry point returns a CI-friendly exit code — 0 = gate passed, 1 = gate failed, 2 = config error — and emits JUnit XML for any CI system.

Reliable exit codes: invoke the gate via python -m ingestion_validation discover … or the compass-discover console script — both sys.exit() the code directly. Avoid piping (… | tee) which returns the pipe's exit code, and note that the compass shim in Git-Bash on Windows can drop the code. To verify in a shell: … ; echo $? (bash) or …; echo $LASTEXITCODE (PowerShell).

Gate strictness (discovery mode) is yours to choose:

# Strictest: any failed document fails the build (default)
compass-discover --project my-kb

# Tolerate a few failures: require >= 95% of documents to pass
compass-discover --project my-kb --fail-under 95

# Also require the corpus to be READY (completeness/coverage/dedup verdict)
compass-discover --project my-kb --require-ready

GitHub Actions — drop the reusable action into your KB pipeline:

# .github/workflows/kb-readiness.yml
jobs:
  kb-readiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4              # your repo holds config/projects/<name>.yaml
      - uses: TestAutomationArchitect/compass@v1
        with:
          project: my-kb
          require-ready: "true"                # block promotion unless READY
          # fail-under: "95"                   # ...or tolerate a bounded failure rate
        env:
          REDIS_HOST: ${{ secrets.REDIS_HOST }}   # config references env-var names; supply them here

A full example you can copy is in examples/kb-readiness.yml.

Container — no Python setup needed; works in any CI or a self-hosted runner:

docker build -t compass:latest .
docker run --rm -v "$PWD:/work" --env-file .env \
  compass:latest compass-discover --project my-kb --require-ready

Applicable Domains

Enterprise document management · legal/compliance · healthcare · financial research · customer-support KBs · internal wikis · e-commerce catalogs · academic repositories — any RAG/agent knowledge base built from a document pipeline.

License

MIT License — see LICENSE.