COMPASS — the quality gate for your RAG knowledge base

Point COMPASS at your RAG knowledge base. It runs 73 deterministic checks across storage, knowledge graph, vectors and retrieval, and fails your CI build when the KB is broken. No LLM, no test-writing — the checks ship with it.

Think of it as a linter for your RAG data plane: a catalogue of built-in checks with HARD/WARN severity tiers, tuned in YAML, gating CI by exit code. It answers "is the knowledge base itself sound?" — not "are the answers good?" (that's the eval layer's job; COMPASS is the gate that runs first).

Your KB is broken and you don't know yet

Chunks that never got embedded. An embedding model swapped without a re-index. A document deleted at the source but still answering questions. Citations pointing at the wrong file. Storage, graph and vector store quietly disagreeing about what exists. All of it is invisible from the outside — retrieval still returns something, answers still look plausible, and your eval scores drift for reasons nobody can trace.

COMPASS finds these deterministically, before your agents serve them.

30 seconds to see it work

pip install compass-kb-validation
compass demo                                   # sample report, no backend needed
compass check redis://localhost:6379/my_index  # your real index: profile -> validate -> report

compass check needs no config: it profiles the index, fills in the shape it detects, runs the checks, and opens an HTML report. See QUICKSTART.md for the five-step path to a gating CI check.

Does it actually catch things?

Yes — and you can verify that claim yourself in one command, no services required:

python benchmark/broken_kb_benchmark.py
#   Detection rate:      16/16  (100%)
#   False-positive rate: 0/6    (0%)

The broken-KB benchmark injects 16 realistic faults (zero chunks, wrong embedding dimension, degenerate vectors, cross-document contamination, orphaned sources, retrieval drift, …) into fixtures, runs the real validators against them, and scores detection vs false positives. It also runs 6 healthy fixtures to prove COMPASS doesn't cry wolf. It's part of the test suite, so the number can't rot.

What COMPASS Does

COMPASS validates a document's journey through your ingestion pipeline and verifies the resulting knowledge base is agent-ready. The pipeline is declarative — you choose which steps run, in what order, in YAML — and ships with these built-in steps:

API Upload → Object Storage → Knowledge Graph → Vector Store → Retrieval → Cross-Stage Consistency

Enable only the steps your setup has (no object store? no graph? drop them), reorder them, or register your own. For each enabled step it deterministically checks, for example:

• Ingestion — the document was accepted and assigned an id (GraphQL or REST ingestion APIs, config-driven)
• Object storage — it persisted with the right content type, size, and integrity, and (opt-in) the source document is healthy across PDF/DOCX/PPTX/HTML (S3, MinIO, R2, Azure Blob, GCS…)
• Knowledge graph — metadata is present, correct, and agrees with the other stores: required predicates, identity uniqueness, storage-reference integrity, RDF typing, ontology-namespace consistency, dangling links, orphan nodes, version supersession (GraphDB, Neptune, Neo4j, Stardog, Blazegraph…) — 19 checks, HARD / WARN tiered, every predicate and query config-driven so they fit any ontology
• Vector store — chunks exist with valid embeddings, correct dimensions/metric, sane ordering/overlap, structure coherence, retrievable via KNN, no cross-document contamination, no near-duplicate bloat (Redis, Qdrant, pgvector, Chroma, Pinecone, Weaviate) — 30 toggleable checks, each on a HARD / WARN severity tier (only HARD gates the run)
• Retrieval — the document's own content is actually retrievable (recall@K, MRR); opt-in embedding-drift, provenance, filtered-recall, negative-retrieval, keyword-completeness
• Cross-stage consistency — do object storage, the graph, and the vector index agree about each document (eTag, metadata, ACL propagation)?

Two modes: live validation as documents are ingested, and post-hoc discovery of a corpus that's already loaded.

Scope note: COMPASS validates pipeline & data integrity. It does not judge answer quality — that's the evaluation layer's job. COMPASS is the gate that runs first.

Key Features

✅ Declarative pipeline — choose/reorder/extend steps in YAML; no fixed shape
✅ Config-Driven — zero code changes per project; all customization in YAML
✅ Pluggable backends — storage / vector store / knowledge graph via a provider registry (one file to add a vendor). Redis, Qdrant, pgvector, Chroma, Pinecone and Weaviate all support metadata pre-filters, so a probe reproduces production's filtered retrieval on any of them (SUPPORT.md)
✅ Pluggable steps — register custom validators in a step registry; the report adapts automatically
✅ Deterministic & cheap — structural/byte/schema checks, no LLM, CI-gateable
✅ Query & analyze the KB — ask it questions, expose it to agents over MCP, and preview embedding-model migrations / map retrieval coverage — all deterministic, read-only (see Beyond validation)
✅ Eval oracle — enrich any eval with deterministic KB-grounding signals, then measure how accurate its LLM judges are against that ground truth (compass enrich / compass calibrate-judges) — the eval framework guesses, COMPASS checks the KB
✅ Interactive reporting — self-contained HTML dashboard + JSON + JUnit XML
✅ Trend Tracking — SQLite historical performance analysis
✅ Notifications — Slack/Teams webhooks for alerts
✅ Parallel Execution — process many documents concurrently
✅ Graceful Degradation — non-critical failures don't crash the pipeline

Install

Core deps only; add extras per backend (Redis is built in):

pip install compass-kb-validation
pip install "compass-kb-validation[qdrant,pgvector,chroma]"
#   vector stores: qdrant | pgvector | chroma | pinecone | weaviate
#   also: pdf (PDF source checks) | content (HTML extraction) | azure | gcs | dashboard

Project path (saved config)

Then use the unified compass CLI - one verb, subcommands:

compass init my-kb          # scaffold config/projects/my-kb.yaml + .env.example + a .compass marker
cp .env.example .env        # fill in your backend connection
#   edit config/projects/my-kb.yaml  (search_index, document_id_field)
compass doctor --project my-kb     # verify connectivity + config (per stage)
compass discover --project my-kb   # validate the already-populated KB, emit a report

compass init drops a .compass marker, so the CLI works from any subdirectory of the project (no env var to set). Other subcommands:

compass validate --project my-kb --output-format all   # live ingestion validation
compass profile  --project my-kb                        # read a store's shape, suggest config
compass list-checks                                     # check catalogue + severity tiers (no project needed)
compass <cmd> --help                                    # per-command options

The standalone compass-discover / compass-validate / compass-profile commands keep working. Live validation also reads a test_data/<name>.yaml (scaffold it with compass-validate --project my-kb --init-test-data).

Beyond validation — inspect, migrate, map coverage

These read-only, deterministic (no-LLM) tools help you verify, diagnose, and safely change a knowledge base — they don't turn COMPASS into a retrieval app or an agent-serving platform. compass ask is a retrieval probe (see what the KB returns for a query, no generated answer); compass mcp is an inspection surface (read-only KB tools an agent can call, not a serving layer). Everything here stays on COMPASS's side of the line and never affects a validation pass/fail. They read an embedding: block in the project YAML (provider openai / azure_openai / bedrock / openai_compatible for a self-hosted server such as vLLM, Ollama or TEI, model_id, dimensions) so a query is embedded the way ingestion was.

# Ask the KB a question - returns the ranked chunks it would retrieve (no generated answer)
compass ask --project my-kb "what is the refund policy?"

# Match production retrieval with metadata pre-filters (hybrid search; commas AND, | ORs a field)
compass ask --project my-kb --filter "category=report,region=US|EU" "summarize the latest report"

# Expose the KB to agents (Copilot/Claude) as MCP tools over stdio
compass mcp --project my-kb --print-config   # prints a .vscode/mcp.json snippet
compass mcp --project my-kb                   # run the server (query_kb, list_documents, ...)

# Preview an embedding-model switch BEFORE you cut over (self-bootstraps, no questions needed)
compass migrate --project my-kb --to openai/text-embedding-3-large

# Map retrieval coverage / find dead zones a question set never reaches (feed real query logs)
compass coverage --project my-kb --questions query_logs.jsonl

# Prove retrieval matches production: compare vs a ground-truth file (precision/recall/Jaccard)
compass compare --project my-kb --ground-truth production_ground_truth.json
# Mixed ground-truth file? Non-retrieval cases (no chunks) are auto-skipped; filter by variant:
compass compare --project my-kb --ground-truth gt.json --variant answerable

# Hand retrieval to any eval framework: emit generic RAG eval test cases
compass ask --project my-kb --batch questions.jsonl --format eval --output eval_input.jsonl

# Corpus health (deterministic, no LLM): score chunk quality / find stale + orphan sources
compass quality   --project my-kb
compass freshness --project my-kb --stale-after 30

# Validate an EXTERNAL eval's output against the KB COMPASS already has access to. Not an eval,
# not a tracer: the eval (an LLM judge, or the bot's captured trace) claims "grounded",
# "cited MIS p.1", "28%" - COMPASS deterministically checks those claims against ground truth
# in the KB and attributes any failure to a layer (retrieval vs the model), never scoring answer
# quality. Verifies each CITED claim at its cited location: a fabricated number, a wrong page, a
# fact present in the KB but not retrieved, or a source not in the KB. Uses the eval's retrieved
# chunks when present (Level 1) and the live KB (Level 2); the answer text + a reachable KB alone
# still catches fabricated figures and phantom citations.
compass grounding-audit --project my-kb --eval-result case.json --fail-on-fabrication

# Level 3 (opt-in): when the KB cannot confirm a cited value, read the ORIGINAL source object and
# check the cited page of the real document. This separates the two cases the KB alone cannot tell
# apart - the fact IS in the source but never reached the KB (an INGESTION defect: re-ingest, no
# amount of retrieval tuning fixes it), versus the fact is nowhere in the source (a PROVEN
# generation hallucination). Needs eval.citation_source_key_fields + object-store access.
compass grounding-audit --project my-kb --eval-results cases.jsonl --verify-source

# Eval oracle (deterministic, no LLM): enrich an eval with KB-grounded signals, then measure how
# accurate the eval's OWN LLM judges are against that deterministic truth. LLM judges GUESS whether
# a response is grounded; COMPASS CHECKS it against the KB - so it can tell you which judge is right.
compass enrich --project my-kb --input eval.json --output enriched.json
compass calibrate-judges --input enriched.json --metric groundedness

# Close the loop: re-decide the eval's PASS/FAIL using deterministic truth instead of judge
# noise, and report every verdict that changed - cases the eval FAILED that are actually
# grounded (rescued), and cases it PASSED that contain a fabrication (demoted). Pure
# arithmetic over the enriched file: no KB, no LLM, no credentials. CI-ready exit codes.
compass quality-gate --input enriched.json --weights weights.json --threshold 0.80

# Before spending judge money: does each ground-truth answer make the key points declared for
# it? A point the reference itself does not make grades every real answer against a bar the
# reference cannot clear - and those failures look like model defects. Checks the GROUND TRUTH
# against itself, never the system's answers. Deterministic, no KB, no LLM.
compass gt-lint --input ground_truth.json --answer-field answer --points-field key_points

# Pre-flight: which cited document names actually resolve to a KB document? An unresolvable
# name makes every claim citing it unverifiable, so finding out from a finished report means
# paying for the run twice. Exits 1 if any name does not resolve.
compass grounding-audit --project my-kb --eval-results eval.json --check-aliases

# Is retrieval the problem, or is the KB? Compare retrieval configurations over the SAME
# validated KB using the eval's own ground-truth chunks - recall / precision / MRR / NDCG /
# hit rate. No LLM: a chunk is relevant because the ground truth says so.
compass benchmark --project my-kb --input eval.json

# Did the source content reach the store? Compares each document with its chunks: body text
# coverage, per-page coverage and numbers, with the passages behind every finding. See
# "Source fidelity" below. Report-only unless --fail-on-findings.
compass fidelity --project my-kb --sample 25 --format html --output reports/fidelity.html

# Is the re-ingested index better? Scores one eval against two ALREADY-BUILT indices and says
# which won on NDCG and on recall. Nothing is re-embedded. Warns when the indices were chunked
# differently, because document + page matching is then invalid (use --match content).
compass simulate --project my-kb --input eval.json --index-a kb_v1 --index-b kb_v2

# Is my chunker the right choice, before paying to re-embed? Rebuilds sampled documents from
# their chunks, re-chunks the same text, and reports mid-thought cuts, table/list splits, sizes,
# fragmentation and "worse on N of M documents". No embedder, no LLM, never a gate. Text rebuilt
# from chunks is lossy and understates what a fixed splitter does to raw tables.
compass chunk-eval --project my-kb --sample 50 --strategy fixed:3000/200 --strategy recursive:3000

# One verdict for the whole KB, composed from artifacts the commands above already wrote. See
# "KB readiness" below. No live connections. Exit 0 READY, 1 otherwise, 2 bad input.
compass readiness --project my-kb      # finds reports/my-kb-<stage>.json; or name each with a flag
# Do two environments hold the same KB? Documents missing either way, schema, e_tags that
# differ, pollution patterns, stale versions. Exits 0 no drift, 1 drift, 2 config error or could
# not compare: an environment that cannot be read is never reported as "no drift".
compass drift --project my-kb --env-a dev --env-b qa

# Audit traceability: does each sampled document still trace to its source? Classifies orphan /
# stale / broken-ref / no-version; a stage that cannot run says why. CSV for the auditor.
compass lineage --project my-kb --sample 200 --format csv --output lineage.csv

# Grounding dropped between runs - what changed? Attributes the regression to the documents
# responsible, ranked by share ("doc X REMOVED -> 40.0 pp across 8 cases"). A drop that no
# changed document explains is reported as unexplained, never spread to make totals reconcile.
compass diff --project my-kb            # or: --baseline run-a.json --current run-b.json

# Incremental discovery: re-validate only what changed (by content, not clock), plus a
# spot-check of the rest. Unchanged documents are reported as NOT re-validated, never as passed.
compass discover --project my-kb --since last-run

Source fidelity

compass fidelity compares two extractions of a document, so it has to tell extraction noise from loss. Both sides are normalised (Unicode, ligatures, hyphenation, markup, and spacing, including gaps inside words), tables are judged by their numbers, and every exclusion is counted. Install a better text layer with pip install "compass-kb-validation[pdf-layout]", or [pdf] for the lower-bound default.

Start report-only, and list what your pipeline drops on purpose until only real findings remain:

eval:
  citation_source_key_fields: ["s3_key"]   # chunk field holding the source object key
  source_extractor: auto                   # auto | pymupdf | pdfplumber | pypdf | pdf_text
  fidelity:
    text_field: text                       # index field holding chunk text (not the eval file's)
    normalization: aggressive              # minimal | standard | aggressive
    match: shingle                         # shingle | exact
    coverage_min: {default: 0.90, quarterly: 0.97}   # keyed by document_type
    prefer_snapshot: true
    figure_regions: snapshot               # snapshot | off: numbers inside figures reported apart
    figure_types: [figure, chart, image]   # snapshot element types that are figures
    lost_passages_allowed: 0               # named lost passages tolerated (snapshot baseline)
    number_format: "1,234.5"               # or "1.234,5" | "1 234,5" | "1'234.5"
  source_exclusions:                       # nothing is excluded that is not listed here
    repeated_on_pages: 5                   # headers and footers
    min_chars: 75                          # small fragments
    patterns: ["(?i)^past performance"]    # by-design notices
    page_roles: {disclaimer: "(?i)^important disclosures"}
    roles: [marginalia, logo, attestation] # read from a snapshot, so they need one
    must_retain: ["material change"]       # overrides every exclusion; missing is a loss

fidelity also runs the other way, chunk by chunk, and each chunk that does not trace is reported by id: WRONG_PAGE, ALTERED, TRUNCATED, OUT_OF_ORDER, UNTRACEABLE, or UNVERIFIABLE where the source has pages with no text layer. A consistent page shift is reported once. Tune it with eval.fidelity.page_tolerance (default 1), page_field, page_end_field and position_field, or turn it off with chunk_integrity: false.

Two outcomes are setup faults, never losses. A document whose chunks have no text in eval.fidelity.text_field is not comparable, and the reason names the fields that do hold text. A document where not one chunk traces to its source is SOURCE_MISMATCH: either the field or the source key is wrong, or the store holds another document's text under that id. Neither is averaged into coverage as a 0%. A snapshot that cannot be found is reported with the key it was looked for at.

What gates is what can be named. Against a snapshot, every passage the extractor produced and the store lacks is a lost passage, reported with its element type, page and box, and one is enough for CONTENT_LOSS (lost_passages_allowed raises that). A number in body text that is not stored is listed with its page and sentence. Numbers inside a figure region, and runs of evenly spaced numbers such as an axis (20.0% 22.0% 24.0%), are reported apart as figure_numbers and never gate; words inside a figure are still compared. Page coverage is shown as context, not as a gate. The JSON keeps every finding, body text first; the console shows the first few. A caption or a Source: note stored more than one chunk away from its figure is CAPTION_DETACHED, with both chunk ids. A source that cannot be read (encrypted, image-only) is compared through its snapshot.

Findings say what is missing, not who lost it, until you declare the extractor's own output. Any extractor that writes JSON works: map its fields once, and each finding becomes PIPELINE_DROP, PIPELINE_ALTER or EXTRACTOR_MISS.

eval:
  snapshot:
    provider: storage_json
    location: "snapshots/{project}/{id}/{e_tag}/document.json"   # also {source_key}
    fallback_locations: ["backup/{e_tag}/document.json"]         # tried in order after it
    version: latest_with_elements          # current (default) | latest_with_elements
    format: my_extractor
    formats:
      my_extractor:                        # paths into the extractor's JSON
        elements: "metadata.element_list"
        type: "chunk_type"
        text: "text"
        order: "chunk_index"
        page: "grounding[0].page"          # [n] picks one item
        boxes: "grounding[*].box"          # [*] takes all: an element may have several boxes
        box_keys: [left, top, right, bottom]   # for boxes written as objects
        page_base: 0                       # the extractor counts pages from 0
        cells: "table.cells"

version: latest_with_elements reads older object versions when a later write replaced the element list with a text-only file, or the key was deleted under bucket versioning. Each document's snapshot is present, superseded (read from an older version or a fallback), missing, wrong_shape or unreadable; the counts are in snapshot_health, and readiness.min_snapshot_coverage (0 to 1) fails readiness when too few were read. compass doctor reads the snapshots of a few documents, says which location and version holds the element list, and prints a formats entry to paste when the mapping is wrong.

KB readiness

compass readiness --discover D.json [--grounding G.json] [--benchmark B.json] [--drift X.json] [--project P] [--format console|json|html] [--output F] reads JSON the other commands wrote and returns READY, AT RISK or NOT READY, with a stage and an owner (ingestion, retrieval config, generator) for every finding. A stage whose artifact is absent, or which measured nothing, is not examined and caps the verdict at AT RISK: never READY by omission. With --project, a stage no flag names is read from readiness.artifacts ({stage: path}), then from reports/<project>-discovery-report.json and reports/<project>-<stage>.json for benchmark, grounding, drift and fidelity. A stage still missing names the command that writes it. gate means NOT READY, risk means AT RISK, advisory is reported and changes nothing.

Every verdict carries verified / unverifiable / not-checked counts, and discover prints and exports the same block (coverage in the JSON). unverifiable means it was attempted and could not be compared, with the reason; not-checked means nobody tried. A fidelity comparison that never ran reads content-fidelity: not-checked, under a READY too when the stage is waived. This is reporting: it changes no verdict.

* A "could not check" finding: it can be raised to gate, never lowered to advisory.

Change the table per project. An unknown key exits 2, because a typo here would turn a gate off:

readiness:
  severity: {stale_versions: risk, grounding_direction_mismatch: gate}
  waived_stages: [drift]     # stages this project does not run, so absence stops capping
  min_recall: 0.6            # 0 = no floor: recall is reported, not gated
  min_kb_confirmed_pct: 80

enrich adds a compass block per case (a citation grounding rate over the verifiable citation+numeric subset, citation fidelity, chunk quality, context utilization, COMPASS_* signal codes). Each citation also carries a graduated support tier — SOUND / LIKELY_SOUND (≥70% of its values found) / PARTIAL / UNSUPPORTED — reported alongside the strict rate, so a near-miss is not filed next to a fabrication. Short cited tokens ("WIU", "Q3 QIP") resolve automatically against the index's own document names; a token owned by two documents resolves to nothing rather than guessing. It is schema-agnostic — every field resolves through your eval.gt_* config, so it works whatever shape your eval JSON takes. calibrate-judges then reports each judge's error, bias, accuracy, and an evidence-based recommended weight — something no eval framework can compute, because it needs KB access.

migrate and coverage also render a self-contained HTML report (same look as the validation dashboard) with --html [path] --open — a coverage heatmap, a migration stability distribution, verdicts, and the most-changed / dead-zone tables.

Same boundary as the gate: these never judge answer quality (that's the eval layer) and never put an LLM in a pass/fail decision. migrate and coverage share one deterministic retrieval-analysis engine (sample → in-memory KNN → neighbor/reachability analysis).

Catch regressions over time (baselines + drift)

Snapshot a known-good run, then diff later runs against it — so a scheduled check is a gate, not just a report:

compass discover --project my-kb --save-baseline          # freeze a green run -> baselines/my-kb.json
# ...later (cron / CI)...
compass discover --project my-kb --against baselines/my-kb.json --fail-on-regression

Drift flags what a single run can't: dropped documents, docs that passed before and fail now (and which checks newly fail), index drift (embedding dimension / distance metric changed under you), and readiness / pass-rate decline. --fail-on-regression makes the run exit non-zero, so cron or CI alerts on degradation. (compass check takes the same --save-baseline / --against flags.)

> From source instead of PyPI: `git clone … && cd compass && pip install -e ".[all]"`.

### Running scenarios without editing YAML

`compass-discover` takes overrides that win over config + profiling, so you can try
different shapes and policies from the command line:

```bash
# Point at a different index/identity, faster, and relax the readiness bar
compass-discover --project my-kb --parallel --workers 8 \
  --id-field agent_id --expected-dim 1024 \
  --readiness-threshold 0.99 \
  --disable-check chunk_count near_duplicate_embeddings \
  --severity duplicate_detection=hard \
  --unreachable-as skipped

Severity model: every vector check is HARD (gates the run) or WARN (advisory). --severity name=hard|warn (or redis.check_severity in YAML) promotes/demotes any check; --list-checks shows the defaults.

Releases are published via PUBLISHING.md.

Reports are written to reports/ (HTML dashboard + JSON + JUnit XML). Add --live to open a real-time progress dashboard, --trends to record history, --parallel for concurrency.

Try it locally (no cloud needed)

A self-contained Docker stack runs the full pipeline against MinIO + Redis Stack + a mock ingestion API — see integration/README.md.

Project Structure

compass/
├── run_validation.py              # Entry point — live ingestion validation
├── discover_and_validate.py       # Entry point — post-hoc discovery
├── compass/          # Main package
│   ├── models/                    # StepResult / PipelineResult (stdlib dataclasses)
│   ├── config/                    # Layered YAML config + typed Settings
│   ├── utils/                     # Run-id, logging, polling w/ backoff
│   ├── providers/                 # Pluggable backends via a registry:
│   │   ├── storage.py             #   object storage (S3-family…)
│   │   ├── vectorstore.py         #   vector store (Redis, Qdrant, pgvector, Chroma…)
│   │   ├── graph.py               #   knowledge graph (SPARQL, Neptune…)
│   │   ├── embedding.py           #   text→vector for query (OpenAI/Azure/Bedrock)
│   │   └── source.py              #   document discovery (storage|index|manifest)
│   ├── validators/                # BaseValidator + step registry + the steps
│   │   └── registry.py            #   declarative pipeline (StepSpec registry)
│   ├── pipeline/                  # Orchestrator (context propagation, halting)
│   ├── report/                    # Self-contained HTML dashboard + JSON/JUnit
│   ├── cli.py                     # Unified `compass` CLI (all subcommands)
│   ├── ask.py                     # `compass ask` — deterministic NL retrieval
│   ├── mcp_server.py              # `compass mcp` — agent-callable KB tools (stdio)
│   ├── retrieval_analysis.py      # engine for `migrate`/`coverage`/`compare` (read-only)
│   ├── eval_interop.py            # vendor-neutral eval export + ground-truth reader
│   ├── corpus.py                  # Corpus-level KB readiness analysis
│   ├── live_dashboard.py          # FastAPI + SSE real-time dashboard
│   ├── notifications.py           # Slack / Teams webhooks
│   └── trend_tracker.py           # SQLite historical trends
├── config/{base.yaml, projects/_template.yaml}
├── test_data/_template.yaml       # Documents to validate (live mode)
├── integration/                   # Local Docker integration stack
├── tests/                         # Unit tests (backends faked — no live services)
└── reports/                       # Generated reports (gitignored)

Documentation

• QUICKSTART.md — pip install → a gating CI check in five steps
• ARCHITECTURE.md — how COMPASS is built: layers, data flow, the registries, the two run modes, extension points
• CONTRACT.md — the Knowledge-Base Contract: express your KB's guarantees as one declarative artifact, with each clause mapped to the command/flag that enforces it in CI
• CASE_STUDIES.md — the reproducible benchmark story in full, plus templates for real-outcome case studies (missing docs, filter regressions, migration drift, orphan chunks)
• benchmark/ — the broken-KB benchmark: injects realistic faults and scores detection rate (currently 16/16) vs false-positive rate (0/6)
• SUPPORT.md — backend support matrix (which backends are verified vs adapter-available)
• COMPATIBILITY.md — stability & versioning policy: what's stable, what's evolving, deprecation & pinning
• EXTENDING-BACKENDS.md — add a new backend (storage/vector/graph/source) or a new pipeline step
• integration/README.md — run the full pipeline locally on Docker
• CONTRIBUTING.md · SECURITY.md

Configuration

All behaviour lives in YAML; credentials are referenced by environment-variable name and resolved at runtime (never stored in config). config/base.yaml holds shared defaults; each config/projects/<name>.yaml overlays only what differs and selects a dev/staging/prod environment block.

# config/projects/my-kb.yaml
display_name: "My Knowledge Base"

# Declarative pipeline — choose which steps run, in what order (registry keys).
# Omit to use the full default pipeline.
pipeline:
  steps: [api_upload, s3_storage, redis_chunks, retrieval_quality]
  critical_steps: ["API Upload", "S3 Storage"]   # failure here halts the run

# Where post-hoc discovery finds documents: storage | vectorstore | manifest
discovery:
  provider: vectorstore           # enumerate the KB straight from the index

environments:
  dev:
    api:
      base_url_env: API_BASE_URL          # env-var NAME, not the value
      graphql_mutation: |
        mutation ($file: Upload!, $metadata: JSON) { uploadDocument(file:$file, metadata:$metadata){ status document_id } }
    s3:
      provider: s3                        # s3 | minio | r2 | azure_blob | gcs | local …
      bucket_name: my-documents
      prefix: knowledge-base/
    redis:
      provider: redis                     # redis | qdrant | pinecone | pgvector …
      search_index: my_embeddings
      text_field: text                    # the field holding chunk text (content, page_content, ...)
      expected_embedding_dim: 1536
      expected_distance_metric: COSINE

redis.chunking_strategy selects the chunk checks. Under anything but fixed, size-derived checks stop applying and advisory checks take over: fragmentation, summary presence, element_integrity (a chunk holding part of a table, list or code block), boundary_on_structure (a boundary inside one, which needs a positional field) and cap_truncation (a chunk at max_chunk_size that ends inside one). A structure continuing onto the next page, a mid-sentence end and a heading left at a chunk's tail are how layout chunkers work, and are not counted. The strategy comes from the chunks when redis.chunker_field and chunker_strategy_map are set, else from chunking_strategy. A project that declares neither is scored exactly as before: where the chunks are clearly not fixed-size and carry a summary or markdown structure, the structural rows appear marked inferred, with their evidence, and are never scored.

embedding.input_fields, input_joiner and input_truncation declare what the pipeline embeds, so the embedding probes re-embed that and nothing else. embedding.sync_check (off by default) has discover re-embed a sample and name chunks whose vector does not match; a chunk missing a declared field is unverifiable, not out of sync, and tokens:N truncation is reported, never estimated.

discover --since re-validates a document when its ETag, its chunk count, or (where the store can list them in one aggregation: Redis today) its chunk ids or redis.chunker_version_field value changed. The id hash is over sorted ids, so it sees a changed set rather than a reordering, and ids built from document + position do not change under a same-count re-chunk: point redis.chunk_id_field at a content hash if the pipeline stores one.

Advanced Usage

Add a custom pipeline step

Steps are declarative — register a validator and reference it in pipeline.steps:

from compass.validators.base import BaseValidator
from compass.validators.registry import register_step, StepSpec
from compass.models import StepResult, StepStatus

class PiiRedactionValidator(BaseValidator):
    step_name = "PII Redaction"
    def __init__(self, config): self.config = config
    def validate(self, context: dict) -> StepResult:
        return StepResult(self.step_name, StepStatus.PASSED, "no PII leaked")

register_step(StepSpec(
    key="pii_redaction", name="PII Redaction",
    factory=lambda settings, shared: PiiRedactionValidator(settings.s3),
    abbr="PII", order=25,
))
# then:  pipeline.steps: [api_upload, pii_redaction, redis_chunks]

The orchestrator, live dashboard, and HTML report adapt automatically. See EXTENDING-BACKENDS.md for adding backends too.

CI/CD integration — the readiness gate

COMPASS is built to run before your agents or eval consume the knowledge base: wire it into the pipeline that publishes the KB and fail the build when the KB is not retrieval-ready. Every entry point returns a CI-friendly exit code — 0 = gate passed, 1 = gate failed, 2 = config error — and emits JUnit XML for any CI system.

Reliable exit codes: invoke the gate via python -m compass discover … or the compass-discover console script — both sys.exit() the code directly. Avoid piping (… | tee) which returns the pipe's exit code, and note that the compass shim in Git-Bash on Windows can drop the code. To verify in a shell: … ; echo $? (bash) or …; echo $LASTEXITCODE (PowerShell).

Gate strictness (discovery mode) is yours to choose:

# Strictest: any failed document fails the build (default)
compass-discover --project my-kb

# Tolerate a few failures: require >= 95% of documents to pass
compass-discover --project my-kb --fail-under 95

# Also require the corpus to be READY (completeness/coverage/dedup verdict)
compass-discover --project my-kb --require-ready

GitHub Actions — drop the reusable action into your KB pipeline:

# .github/workflows/kb-readiness.yml
jobs:
  kb-readiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4              # your repo holds config/projects/<name>.yaml
      - uses: TestAutomationArchitect/compass@v1
        with:
          project: my-kb
          require-ready: "true"                # block promotion unless READY
          # fail-under: "95"                   # ...or tolerate a bounded failure rate
        env:
          REDIS_HOST: ${{ secrets.REDIS_HOST }}   # config references env-var names; supply them here

A full example you can copy is in examples/kb-readiness.yml.

Container — no Python setup needed; works in any CI or a self-hosted runner:

docker build -t compass:latest .
docker run --rm -v "$PWD:/work" --env-file .env \
  compass:latest compass-discover --project my-kb --require-ready

Applicable Domains

Enterprise document management · legal/compliance · healthcare · financial research · customer-support KBs · internal wikis · e-commerce catalogs · academic repositories — any RAG/agent knowledge base built from a document pipeline.

License

MIT License — see LICENSE.