RAGAssure

Test RAG quality. Verify access. Validate evidence. Gate releases.

RAGAssure is a small, provider-neutral contract-testing and release-assurance framework for RAG systems.

It covers the normal things you want from a RAG evaluation harness — golden datasets, retrieval precision/recall, required facts, forbidden evidence, latency, cost, reports, and CI exit codes — and adds one enterprise question that a generic quality score does not answer:

Did this principal retrieve evidence they were actually allowed to use?

Status: pre-release 0.1.0 (alpha). The deterministic contract engine is implemented. Semantic-evaluator integrations, the report UI, and ontology/GraphRAG assertions are roadmap items.

Why this exists

A RAG system can be relevant and grounded and still be wrong for the person asking.

An employee asks about executive compensation. A restricted compensation document may be the most relevant source in the corpus. Relevance does not make it authorized.

The application has to enforce that rule before generation. RAGAssure gives the delivery pipeline a repeatable way to prove the observable result still matches the rule after prompts, models, data, retrieval logic, or policy change.

That is the center of the project: an executable access-and-quality contract, not just another score.

Quickstart

Requires Python 3.11+.

RAGAssure is not published to PyPI yet, so the current install path is from source:

python -m venv .venv
# activate the environment
pip install -e .[dev]

Validate a contract dataset:

ragassure validate datasets/examples/support_faq.yaml

Run against recorded answers without calling a model:

ragassure run datasets/examples/support_faq.yaml \
  --mock datasets/examples/mock_answers.json \
  --min-pass-rate 0.9

Try the deliberately leaky baseline:

ragassure run datasets/examples/support_faq.yaml \
  --mock datasets/examples/mock_answers_leaky.json \
  --min-pass-rate 0.9

A failed contract or configured release threshold returns exit code 1, so a normal shell step can block promotion in GitHub Actions, Jenkins, GitLab CI, Azure DevOps, or another CI/CD system.

See docs/installation-and-ci.md for Docker, live-service integration, and pipeline examples.

A contract is still a normal RAG test

dataset: enterprise-procurement
version: 1
cases:
  - id: finance-approval-threshold
    question: "What approval is required for a $60,000 vendor purchase?"

    principal:
      tenant: acme
      department: finance
      roles: [finance-analyst]

    expected_document_ids:
      - finance-procurement-policy-v4
    min_precision_at_k: 0.50

    forbidden_document_ids:
      - executive-compensation-2026
      - other-tenant-procurement-policy

    required_facts:
      - "$50,000"

    forbidden_answer_contains:
      - "executive bonus"

    budgets:
      max_latency_ms: 1500
      max_cost_usd: 0.02

The caller context travels with the same quality expectations you would already test.

What is implemented now

Quality

• YAML golden datasets
• expected document assertions
• forbidden document assertions
• required facts / answer content
• forbidden answer content
• recall@k
• precision@k, with an optional per-case minimum threshold
• mock / recorded-answer runs
• live-service adapter boundary

Identity-aware assurance

• first-class Principal
• subject, tenant, roles, department, and custom attributes
• principal-to-adapter filter mapping
• forbidden-evidence failures across the retrieved evidence set
• document metadata policy seed for tenant/classification metadata

Operations and delivery

• per-case latency and estimated-cost budgets
• Markdown and JSON reports
• run-level pass-rate, mean-precision, latency, and cost gates
• CI-friendly non-zero exit codes
• Docker support
• provider-neutral Python contracts

The deterministic core does not require a model SDK, vector database, orchestration framework, or LLM-as-judge.

Where it fits

RAGAssure belongs in the delivery path, not the production request path.

Your RAG application still owns:

principal
  -> API / app
  -> identity + authorization
  -> retrieval
  -> allowed evidence
  -> generation
  -> answer + citations
  -> runtime observability

RAGAssure owns the release check:

code / prompt / model / data / policy change
  -> CI/CD
  -> normal tests
  -> start system under test
  -> RAGAssure contracts
  -> report
  -> PASS promote / FAIL block

The framework does not enforce runtime authorization. It verifies that the observable behavior under a known principal still matches the contract.

Connect your own RAG system

The adapter boundary stays intentionally small:

from ragassure import RAGAnswer

class MyRAG:
    def ask(self, question: str, filters: dict[str, str]) -> RAGAnswer:
        ...

Return the answer, retrieved chunks/evidence, and whatever latency, token, or estimated-cost metadata is available. RAGAssure owns the evaluation and release verdict.

The system under test can be:

• a local Python RAG pipeline
• an HTTP/FastAPI service through an adapter
• a cloud-hosted RAG application
• recorded output for deterministic offline CI

A standard-library HTTP example is included at examples/http_adapter.py.

The demo that explains the project

The companion secure-enterprise-ai-assistant application demonstrates the runtime boundary. Its current development milestone uses synthetic data and simulated employee/manager roles so the authorization behavior is deterministic and easy to test while the app evolves toward real semantic retrieval, grounded generation, identity, telemetry, and deployment.

For the question:

What is the executive bonus structure?

The restricted document can be relevant for both users, but the allowed evidence changes with the principal.

RAGAssure replays the case and checks the observable evidence under the appropriate contract. A deliberately leaky fixture turns the access regression into a release failure.

The two repositories stay separate on purpose:

• secure assistant — runtime retrieval, authorization, generation boundary, API, UI, and traces
• RAGAssure — reusable contracts, deterministic evaluation, evidence, reports, thresholds, and release decisions

Use it in any CI/CD system

RAGAssure does not need a dedicated CI plugin. The integration contract is the process exit code.

exit 0      -> contract suite passed -> continue
non-zero    -> contract or release gate failed -> stop promotion

The included GitHub Actions workflow proves both directions: a known-good suite passes and a deliberately leaky suite is expected to fail.

See docs/installation-and-ci.md for example shell stages.

Deterministic first, semantic signals second

Faithfulness, answer relevance, semantic similarity, and LLM-as-judge can be useful.

RAGAssure does not need to reimplement every mature evaluator. The direction is to allow Ragas, DeepEval, or custom evaluators to contribute optional semantic signals alongside the deterministic evidence.

The design rule is:

Deterministic gates first. LLM signals second.

Hard access or policy assertions should remain repeatable and explainable.

Future scope

The project is intentionally small, but there is a clear extension path.

Planned, not implemented yet:

• reusable optional Ragas / DeepEval adapters
• prompt/model/corpus/retriever/policy version metadata
• baseline-versus-candidate evidence diffs
• a small report viewer with principal-matrix views
• audit-bundle export
• semantic entity/relationship assertions where GraphRAG or a knowledge graph genuinely makes a contract more precise

A larger estate can keep contracts by application, domain, tenant boundary, or risk class and run small PR suites plus broader release/nightly suites in parallel. The core can remain stateless while CI/CD does the orchestration.

Project boundaries

RAGAssure is not:

• a chatbot or end-to-end RAG application
• a vector database
• model hosting
• an identity provider
• runtime authorization enforcement
• a generic observability platform
• an ontology or graph database
• a required hosted control plane

The application owns runtime behavior. RAGAssure owns repeatable release evidence.

Documentation

• docs/architecture.md — runtime vs delivery boundaries
• docs/schema.md — executable contract schema
• docs/how-ragassure-works.md — evaluation flow and failure semantics
• docs/installation-and-ci.md — source install, Docker, adapters, and CI/CD
• docs/ui-and-demo.md — report UI direction
• docs/semantic-assurance.md — later semantic assertion boundary
• docs/decisions/0001-eval-first.md — deterministic-first decision record

License

MIT