enterprise-rag-patterns

Authorization behavior and upgrade guidance: see migration and verification notes.

Python retrieval policies and adapters for applications that need to restrict which records enter an AI prompt.

Use this package when you need framework-independent policy objects, multiple regulatory-domain reference patterns, or adapters for an existing RAG application. For example, an advising assistant can filter retrieved student records by identity, institution and authorized record category before constructing its prompt.

Input → output: the core FERPA policy takes an authenticated application scope and document dictionaries with identity/category fields, then returns the permitted dictionaries. Audit records are created separately. The quick start runs without a model API key; real SDK examples in tests show local retrieval and recorded model-input checks. See the API reference for the wider surface and migration notes for strict metadata rules.

Choose the package: use ferpa-haystack for dedicated Haystack Document components. Use this repository for the broader policy and adapter collection. The portfolio guide explains how these packages fit together.

Limits: your application must authenticate the caller, supply trustworthy authorization scopes and ingestion metadata, and route only filtered results into prompts. Public classification is an explicit trusted ingestion decision; missing private metadata is denied. These controls support access-policy implementation; installing the package does not establish legal compliance or secure an entire application.

The problem this solves

Standard RAG implementations retrieve documents and pass them directly to an LLM — with no enforcement of who is allowed to see what. In regulated environments this creates a structural compliance failure: a student receives another student's financial aid record; a patient's ePHI surfaces in an unrelated clinical query; a grid operator's chatbot leaks BES Cyber System documentation to an unauthorized contractor.

This library provides identity-scoped filters, domain policy patterns and structured audit records. Connect the filter before prompt construction and use only its output; a bypassing application path remains outside that protection.

Architecture: Defense-in-Depth

User Query
    │
    ▼
┌─────────────────────────────────────────────────┐
│         Vector Store Retrieval (candidate docs)  │
└────────────────────┬────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────┐
│  LAYER 1 — Identity / Access Gate               │
│  Who is the user? What are their clearances?    │
└────────────────────┬────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────┐
│  LAYER 2 — Regulatory Domain Gate               │
│  Does this document belong to this regulation?  │
└────────────────────┬────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────┐
│  LAYER 3 — Contextual Policy Gate               │
│  Is this access appropriate in this context?    │
└────────────────────┬────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────┐
│  LAYER 4 — Sector-Specific Gate                 │
│  Are there additional domain requirements?      │
└────────────────────┬────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────┐
│  Audit Record (written before LLM sees docs)    │
│  user_id · document_id · regulation_citation    │
│  layers_passed · layers_denied · timestamp      │
└────────────────────┬────────────────────────────┘
                     │
                     ▼
                 LLM Context

Why pre-filter, not post-filter? Post-filtering is a UI concern — the LLM has already processed the unauthorized record. FERPA, HIPAA, and NERC CIP require that disclosure not occur, not that unauthorized data be hidden after the fact. See docs/adr/001-pre-filter-not-post-filter.md.

Installation

pip install enterprise-rag-patterns

With vector store or framework extras:

pip install 'enterprise-rag-patterns[langchain]'
pip install 'enterprise-rag-patterns[haystack]'
pip install 'enterprise-rag-patterns[llama-index]'
pip install 'enterprise-rag-patterns[dspy]'
pip install 'enterprise-rag-patterns[pinecone]'
pip install 'enterprise-rag-patterns[weaviate]'
pip install 'enterprise-rag-patterns[qdrant]'
pip install 'enterprise-rag-patterns[chromadb]'

Quick start

from enterprise_rag_patterns.compliance import (
    StudentIdentityScope, RecordCategory,
    FERPAContextPolicy, DisclosureReason,
)

scope = StudentIdentityScope(
    student_id="stu_001",
    institution_id="univ_abc",
    requesting_user_id="advisor_007",
    authorized_categories={RecordCategory.ACADEMIC_RECORD},
    disclosure_reason=DisclosureReason.SCHOOL_OFFICIAL,
)
policy = FERPAContextPolicy(scope=scope)

# Synthetic retrieval results use top-level fields for this core policy.
retrieved_docs = [
    {"content": "GPA: 3.85", "student_id": "stu_001",
     "institution_id": "univ_abc", "category": "academic_record"},
    {"content": "Another student's record", "student_id": "stu_002",
     "institution_id": "univ_abc", "category": "academic_record"},
]

# Filter before constructing the model prompt.
safe_docs = policy.filter_retrieved_documents(
    retrieved_docs,
    student_id_field="student_id",
    institution_id_field="institution_id",
    category_field="category",
)

assert [doc["content"] for doc in safe_docs] == ["GPA: 3.85"]

# Create an audit record; your application must persist it.
audit = policy.record_access(categories_accessed={RecordCategory.ACADEMIC_RECORD})

Browse the example catalog for sector patterns; individual framework examples may require optional dependencies and provider configuration.

Measured retrieval boundary

From a repository checkout, install pip install -e '.[haystack]', then run:

HAYSTACK_TELEMETRY_ENABLED=false python -m benchmarks.retrieval_boundary --repeats 5 > boundary-results.json
# Optional real BM25 corpus-size experiments:
HAYSTACK_TELEMETRY_ENABLED=false python -m benchmarks.retrieval_boundary --corpus-size 1000 --repeats 5
HAYSTACK_TELEMETRY_ENABLED=false python -m benchmarks.retrieval_boundary --corpus-size 10000 --repeats 5

The benchmark measures this repository's Haystack adapter in a real in-memory BM25 retriever → filter → prompt builder → recording generator route. It compares native attribute filters, the package gate, both together and a required bypass negative control. The local synthetic fixture records its canonical origin and explicitly permitted IDs. There is no cross-repository runtime dependency or model API call.

Every protected run must exclude unauthorized document IDs and content and retain all permitted fixture documents (authorized recall 1.0); every bypass run must expose a seeded canary. A deny-all gate fails the utility check. Native null comparisons admit the fixture's null-public record: this documents a limitation of that predicate, not an inability to implement authorization in Haystack. JSON records authorized recall, unauthorized counts/IDs, retrieval candidate counts, repeated pipeline timings, environment, runtime and installed-distribution versions, fixture hash and benchmark/adapter source hashes. One warm-up is excluded. Async timing includes event-loop startup; store and pipeline construction are excluded. These small synthetic measurements do not establish answer quality or production performance.

The boundary tests also recheck reused candidates against an explicitly changed application permission snapshot. They do not test provider-cache invalidation or external revocation. The benchmark uses an explicit category allowlist: the Haystack adapter accepts string categories, while the core FERPA policy rejects unknown enum values. An async pipeline schedules the adapter's synchronous filter; this is not a new asynchronous policy backend. Existing LangChain and LlamaIndex tests remain separate coverage, not benchmarked comparisons.

For a standalone Haystack tutorial, see the sibling ferpa-haystack example.

Example catalog — 50 patterns

Agentic & Runtime Security (2025–2026 Standards)

| 49 | 49_owasp_llm_rag_security.py | Agentic / RAG Security | OWASP LLM Top 10 2025 — LLM01PromptInjectionFilter (direct + indirect injection pattern matching + instruction override detection), LLM08EmbeddingWeaknessFilter (poisoned vector payload + tampered embedding source blocking), LLM06SensitiveDisclosureFilter (PII + credentials + classification markers DLP before retrieval context), RAGOutputValidationFilter (structured output integrity + hallucination marker + citation format validation before delivery) | | 50 | 50_rag_security_auditor.py | Enterprise Security Audit | Holistic RAG security gap-analysis framework — RAGSystemConfig (27-field configuration snapshot), RAGSecurityAuditor (22 controls across 6 domains: Input Validation, Vector Store Security, Retrieval Controls, Output Security, Action Gating, Observability), RAGAuditReport (scored 0–100, CSA ATF maturity: Sandbox/Controlled/Trusted/Autonomous), framework refs: OWASP LLM/ASI, NIST AI 600-1, ISO 42001, MITRE ATLAS v5.1, CSA ATF, SOC 2, HIPAA, GDPR, FERPA |

Regulation coverage matrix

How this compares to alternatives

No other open-source library enforces regulatory compliance at the pre-retrieval layer across this breadth of frameworks and regulations:

Key architectural distinction: Standard RAG architectures apply compliance logic after the LLM processes documents. Under FERPA's disclosure rule (34 CFR § 99.30) and HIPAA's Minimum Necessary Rule (45 CFR § 164.502(b)), a document in the LLM context window constitutes a disclosure — regardless of whether the output is filtered afterward. This library enforces access control before retrieval, at the vector store query level.

Framework and vector store integrations

Repository structure

src/enterprise_rag_patterns/
├── compliance.py               # FERPA identity scoping + 34 CFR §99.32 audit
├── context.py                  # Multi-source ContextEnvelope assembly
├── session.py                  # Cross-channel session continuity
├── policy.py                   # Escalation + action-boundary policy objects
├── async_compliance.py         # Async wrappers (asyncio / FastAPI)
├── regulations/
│   ├── gdpr.py                 # GDPR Article 17 right-to-erasure
│   ├── hipaa.py                # HIPAA ePHI minimum-necessary + audit
│   ├── iso27001.py             # ISO/IEC 27001:2022 ISMS CBAC
│   ├── nist_ai_rmf.py          # NIST AI RMF 1.0 + AI 600-1
│   ├── owasp_llm.py            # OWASP LLM Top 10 (2025) LLM01/LLM02
│   ├── pci_dss.py              # PCI DSS v4.0 CHD/PAN masking
│   └── soc2.py                 # SOC 2 Type II CC6.1/CC6.6/C1.1
├── vector_stores/
│   ├── pinecone_adapter.py
│   ├── weaviate_adapter.py
│   ├── qdrant_adapter.py
│   └── chroma_adapter.py
└── integrations/
    ├── langchain.py
    ├── langchain_lcel.py
    ├── llama_index.py
    ├── haystack.py
    └── maf.py
examples/                       # 30 runnable sector examples (see catalog above)
tests/                          # 924 passing tests (2 skipped)
docs/
├── architecture.md
├── adr/                        # Architecture Decision Records
│   ├── 001-pre-filter-not-post-filter.md
│   └── 002-two-layer-enforcement.md
└── implementation-note-*.md

Published notes and articles

• Implementation Note 01 — Cross-channel continuity and compliance scope
• Implementation Note 02 — FERPA boundaries in retrieval-augmented generation
• ADR 001 — Why pre-filter, not post-filter
• ADR 002 — Two-layer enforcement: vector store + policy

Near-term roadmap

• Async vector store adapters for FastAPI/asyncio
• ferpa-haystack Haystack Hub listing (pending deepset review)
• LlamaIndex workflow integration for FERPA + HIPAA multi-regulation pipelines
• Google ADK retrieval guard (pre-retrieval enforcement for ADK agents)

Contributing

Read CONTRIBUTING.md and GOVERNANCE.md. Run pytest tests/ -v before opening a pull request.

Citation

@software{rana2026erp,
  author  = {Rana, Ashutosh},
  title   = {enterprise-rag-patterns: Cross-industry compliance patterns for RAG pipelines},
  year    = {2026},
  version = {0.46.0},
  url     = {https://github.com/ashutoshrana/enterprise-rag-patterns},
  license = {MIT}
}

Part of the enterprise AI patterns trilogy

License

MIT — see LICENSE.