SentinelRAG

Triple-gate RAG for CI. Retrieval can look fine in a notebook while a PR quietly tanks recall, ships an ungrounded answer, or lets a poisoned doc into the index. SentinelRAG treats those as three separate checkpoints — and fails the build when any one of them slips — with evidence your existing CI already understands (JSON, JUnit, SARIF).

pip install sentinelrag · sentinelrag-gate CLI · FastAPI + Qdrant reference app · offline mocks in CI · optional Gemini live path

Three checkpoints before merge

Foreign eval scores (promptfoo, Ragas, DeepEval) can feed the same evidence schema — SentinelRAG does not replace those tools; it unifies their numbers with its own gates.

flowchart LR
  A[Corpus change] --> R{Retrieve}
  R -->|pass| G{Ground}
  R -->|fail| X[Fail CI]
  G -->|pass| D{Defend}
  G -->|fail| X
  D -->|pass| E[Evidence · JUnit · SARIF]
  D -->|fail| X

Gate runs offline (no Gemini / GPU / Qdrant required). Optional live stack: FastAPI + Qdrant + bge-small + Gemini for citations.

See it query a corpus

Walkthrough: docs/DEMO.md

Install & run the gate

pip install sentinelrag
# from source:
# uv sync --extra dev

uv run sentinelrag-gate run --refresh
uv run sentinelrag-gate demo-poison
uv run sentinelrag-gate scan data/public_corpus --fail-on high

Outputs: .sentinelrag/gate/ · Policy: config/gate_policy.yaml

Plug into GitHub Actions

- uses: prakash888kp/SentinelRAG/.github/actions/sentinelrag-gate@v0.2.1
  with:
    refresh: "true"

Bring scores from other evaluators

uv run sentinelrag-gate import promptfoo examples/import_promptfoo/results.json -o build/art.json
uv run sentinelrag-gate import ragas examples/import_ragas/evaluation-result-records.json \
  --metric faithfulness --metric answer_relevancy -o build/art.json
uv run sentinelrag-gate import deepeval examples/import_deepeval/report.json -o build/art.json

Recipes: CI cookbook · Mapping: Interoperability

In the box

• App — FastAPI ingest/query, hybrid + agentic retrieval, complexity router
• Gate — policy YAML, baselines, SARIF/JUnit, composite Action
• Security loop — hygiene, provenance SHA, poison benchmark v1
• Interop — promptfoo / Ragas / DeepEval → sentinelrag-evidence-v1
• Schemas & examples — schemas/ · examples/

Benchmarks

Committed JSON + plots in benchmarks/. Scripts: docs/BENCHMARK.md.

At a glance

Quality vs cost

Hybrid hits full recall cheaper than agentic; naive is cheapest but weaker.

Poison probes vs hybrid α

Ingest defenses stay at 1.0 blocked. Raw retrieval ASR only drops when α → dense (1.0).

Retrieval detail

Raw files: phase1_retrieval*.json, phase2_eval.json, phase3_trajectory.json, pareto_points.json, asr_vs_alpha.json, poison_benchmark_v1.json, hygiene_report.json, baseline_evidence.json.

Dig deeper

Architecture · Metrics · Benchmarks · Deployment · Poison v1 · OWASP map · Story · Phases · Comparison · Changelog

License

MIT · NOTICE