llm-shield-proxy 1.0.3

Enterprise Zero-Egress Privacy Redaction Proxy Engine for LLMs

LLM-Shield-Proxy - Enterprise Privacy Redaction Engine

A zero-latency, zero-egress, streaming-safe PII redaction proxy for Enterprise LLMs.

LLM-Shield-Proxy is an open-source, zero-egress middleware proxy that intercepts OpenAI-compatible LLM API requests, redacts Personally Identifiable Information (PII) before it leaves your local infrastructure, and deterministically re-hydrates real-time SSE streaming responses without breaking stream latency.

Designed to unblock enterprise privacy compliance (SOC 2 / HIPAA).

Author & Core Maintainer: Ninad Phalak ([e-mail])

---

🏗️ Architecture & Data Flow

flowchart TD
    classDef client fill:#e0f2fe,stroke:#0284c7,stroke-width:2px,color:#0369a1,font-weight:bold;
    classDef proxyEngine fill:#f8fafc,stroke:#475569,stroke-width:2px,color:#0f172a,font-weight:bold;
    classDef piiSecurity fill:#fef2f2,stroke:#ef4444,stroke-width:2px,color:#991b1b,font-weight:bold;
    classDef vault fill:#fffbebe,stroke:#f59e0b,stroke-width:2px,color:#92400e,font-weight:bold;
    classDef upstream fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8,font-weight:bold;

    UserApp["👤 User Application\n(OpenAI / LangChain SDK)"]:::client

    subgraph SecurityMoat ["🛡️ Zero-Egress Local Environment (Apache 2.0 Licensed)"]
        direction TD
        FastAPIProxy["⚡ FastAPI Catch-All Proxy\n(/{path:path})"]:::proxyEngine

        subgraph CascadeEngine ["🔒 Two-Tier PII Cascade Engine"]
            Tier1["Tier 1: Compiled Regex"]:::piiSecurity
            Tier2["Tier 2: Quantized ONNX NER"]:::piiSecurity
            Tier1 --> Tier2
        end

        VaultStore[("🔑 Session Vault Store\n(Deterministic Tokens)")]:::vault
        LookaheadBuffer["⏱️ Sliding-Window Lookahead Buffer\n(Prevent SSE Tag Leaks)"]:::proxyEngine
        Rehydrator["🔄 Stream Re-hydrator\n(Token -> Original Value)"]:::proxyEngine
    end

    UpstreamLLM["☁️ Upstream LLM Provider\n(OpenAI / Anthropic / vLLM)"]:::upstream

    %% Inbound Flow (Prompt Sanitization)
    UserApp -- "1. Inbound Raw Prompt Payload" --> FastAPIProxy
    FastAPIProxy -- "2. Scan Payload" --> Tier1
    Tier2 -- "3. Store Vault Keys" --> VaultStore
    Tier2 -- "4. Redacted JSON Payload" --> UpstreamLLM

    %% Outbound Flow (Streaming De-redaction)
    UpstreamLLM -. "5. Raw SSE Stream Deltas" .-> LookaheadBuffer
    LookaheadBuffer -- "6. Tag-Safe Assembly" --> Rehydrator
    Rehydrator <--> VaultStore
    Rehydrator -. "7. Sanitized Real-Time Stream" .-> UserApp

    style SecurityMoat fill:#f8fafc,stroke:#0284c7,stroke-width:2px,stroke-dasharray: 5 5,color:#0f172a
    style CascadeEngine fill:#ffffff,stroke:#cbd5e1,stroke-width:1px

How It Works (The Data Flow)

📥 Inbound (Prompt Sanitization)

1. Intercept: Your application sends a standard OpenAI / LangChain payload to localhost:8000.
2. Cascade Redaction: The proxy intercepts the JSON and routes text through a high-speed compiled Regex engine (SSNs, emails, credit cards), falling back to a local ONNX model for unstructured names.
3. Vault Storage: The original PII is mapped to a deterministic tag (e.g., [PERSON_1]) and stored locally in a TTL-backed session vault.
4. Clean Egress: A 100% sanitized payload is forwarded to OpenAI. OpenAI never sees your raw sensitive data.

📤 Outbound (Streaming De-redaction)

1. SSE Stream Intercept: OpenAI streams the response back chunk-by-chunk via Server-Sent Events (SSE).
2. Lookahead Buffer: Because tags can be split across SSE chunks (e.g., [PER in chunk N and SON_1] in chunk N+1), the proxy's sliding-window buffer holds back unclosed brackets to prevent tag leaks.
3. Re-hydration: Once a tag is fully assembled, the proxy swaps the real data back from the local vault and streams the final, un-redacted text to the user's application in real-time.

---

🔒 Why LLM-Shield-Proxy? (Architectural Moats)

Why not just write a basic regex script? Basic regex scripts break on streaming responses, leak split tokens, and slow down your application. LLM-Shield-Proxy is purpose-built for production:

• Streaming Safety: Includes a proprietary Sliding-Window Lookahead Buffer that prevents Server-Sent Event (SSE) chunking leaks across split tokens (e.g., holding back [PER until SON_1] arrives).
• Speed & Accuracy: Leverages a Two-Tier Cascade Engine (sub-millisecond compiled Regex + Quantized ONNX NER) that catches unstructured names in ~10ms with zero cloud dependencies.
• Context Preservation: Uses a Local TTL Session Vault that maps PII to session-bound tokens ([PERSON_1]), enabling the LLM to retain full conversational context without receiving raw sensitive data.
• Zero-Egress Security: Runs 100% on your local infrastructure. Your raw data never leaves your server; only sanitized, redacted payloads reach upstream LLM providers.

---

⚡ Core Features

• Zero Latency Streaming: Sliding-window tag-safety buffer intercepts SSE streams delta-by-delta without buffering full requests or responses.
• Zero Cloud / Zero Egress: 100% local processing. No external API calls for PII detection.
• Two-Tier PII Cascade Engine:
  • Tier 1 (Sub-millisecond Regex): SSNs, Credit Cards, Email Addresses, Phone Numbers, IPv4/IPv6, API Keys.
  • Tier 2 (NER Engine): Person Names and unstructured entities.
• Deterministic Re-Hydration Vault: Swaps PII with session-bound tokens (e.g., Sarah -> [PERSON_1]). Maps back deterministically when the LLM streams responses. Supports request-scoped and session-scoped (X-Session-ID) vaults.
• SOC 2 Structured Audit Logging: Emits JSON structured audit logs for compliance monitoring.
• Tier 2 (NER Engine): A lightweight, quantized ONNX Named Entity Recognition (NER) model. It uses local AI to catch unstructured names and entities that slip past standard regex, executing in ~10ms without requiring heavy Python frameworks or cloud APIs.

---

🛠️ Quickstart

Installation

Install the package from PyPI:

pip install llm-shield-proxy

Configuration

Create a .env file in the root directory before starting the server.

# Required upstream provider key (or pass via Authorization header)
OPENAI_API_KEY=sk-your-openai-key-here

# Optional configuration
PORT=8000
TELEMETRY_ENABLED=false

1. Start the Proxy

Run the proxy locally via Docker or Uvicorn. No external database required for the open-source core.

uvicorn app.main:app --host 0.0.0.0 --port 8000

or via Docker Compose:

docker-compose up -d

2. Update your Application (1-Line Change)

Point your existing OpenAI SDK base_url to your local LLM-Shield-Proxy instance.

from openai import OpenAI

client = OpenAI(
    api_key="your-openai-api-key",
    base_url="http://localhost:8000/v1" # Point to LLM-Shield-Proxy
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "user", "content": "Contact Sarah Connor at [e-mail]"}
    ],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

---

📊 Performance & Memory Benchmarks

LLM-Shield-Proxy is engineered for sub-millisecond overhead and ultra-lightweight resource usage. Measured over 1,000 production streaming iterations:

Metric	Average Latency	Median Latency	Footprint / Notes
Tier 1 Regex Overhead	0.0294 ms	0.0291 ms (29.10 µs)	Microsecond pattern scan
Tier 2 NER Overhead	0.0033 ms	0.0032 ms (3.20 µs)	Quantized local NER scan
Total SSE Stream Overhead	0.0010 ms	0.0010 ms (0.97 µs)	Added latency per SSE delta chunk
Process RAM Footprint	-	-	24.55 MB Resident Set Size

To run the automated benchmark suite locally:

py tests/benchmark.py

---

⚠️ Known Limitations

Transparency is critical for security tooling. Please be aware of the following current limitations:

• Text Only: The proxy does not currently scan or redact text embedded inside base64 image payloads (e.g., OpenAI Vision models).
• Supported Languages: The Tier-2 ONNX NER model is currently optimized for English-language entities.
• Non-Standard Streaming: Designed for standard Server-Sent Events (SSE). Custom or proprietary streaming protocols may bypass the sliding-window buffer.

---

🧪 Testing

Run the full automated test suite:

py -m pytest tests/

---

🚀 Enterprise Deployment & Operations

Designed for zero-friction adoption by DevOps, Site Reliability Engineers (SREs), and Network Administrators:

1. 🏥 Health Check Endpoints (Kubernetes & Swarm Probes)

Built-in liveness and readiness endpoints return HTTP 200 OK for Kubernetes, Docker Swarm, or AWS ECS health monitors:

curl http://localhost:8000/health
# Output: {"status":"ok","service":"llm-shield-proxy","version":"1.0.3"}

curl http://localhost:8000/livez
# Output: {"status":"ok","service":"llm-shield-proxy","version":"1.0.3"}

2. ⚙️ 12-Factor Environment Configuration

100% compliant with 12-factor app standards. All upstream target routing and API keys are injected via environment variables or a .env file without code modifications:

• UPSTREAM_BASE_URL: Base target URL (e.g. https://api.openai.com or internal vLLM server).
• OPENAI_API_KEY: Upstream API key passed to target providers.
• REDIS_URL: Optional Redis connection string for distributed multi-instance session caching.

3. 📈 Stateless & Horizontal Scaling

LLM-Shield-Proxy runs completely stateless by default. For high-volume enterprise deployments, instances scale horizontally behind edge proxies (NGINX, Traefik, AWS ALB):

docker-compose up -d --scale proxy=5

When configured with REDIS_URL, session vaults are shared across all proxy replicas, ensuring seamless session isolation across multi-instance clusters.

4. 🔒 Supply Chain Integrity & GPG Signature Verification

Every published release includes automated SHA-256 checksums (checksums.txt) and GPG detached signatures (checksums.txt.asc) signed by maintainer Ninad Phalak. You can verify checksums and cryptographic authenticity before deployment using:

# 1. Verify SHA-256 Checksums (Linux / macOS):
sha256sum -c checksums.txt

# On Windows (PowerShell):
Get-FileHash llm-shield-proxy-source-v1.0.3.zip -Algorithm SHA256

# 2. Verify Cryptographic GPG Signature:
gpg --verify checksums.txt.asc checksums.txt

---

🌍 Internationalization (i18n) & GDPR Roadmap

Currently, LLM-Shield-Proxy's Tier 1 Regex engine is optimized for North American PII (US SSNs, Phone Formats). To support global GDPR compliance, we are actively looking for contributors to help expand our regex payloads and Tier 2 ONNX models for:

• European Formats: UK NIN, EU Phone Numbers, IBANs.
• APAC Data Structures: India Aadhaar, APAC localized identifiers.
• Multilingual NER ONNX Models: Multilingual entity recognition models.

If you want to contribute to enterprise AI security, check out our CONTRIBUTING.md and claim a locale!

---

🗺️ Future Technical Roadmap (Performance & Scale)

We are committed to maintaining LLM-Shield-Proxy as the fastest zero-latency redaction engine for LLMs. Here are the core architectural optimizations planned for upcoming releases — contributions and PRs are warmly welcomed:

1. ONNX Thread Tuning (Preventing CPU Contention)

   • Problem: By default, ONNX Runtime attempts to use every available CPU core. In FastAPI, this competes with the event loop handling thousands of concurrent connections.
   • The Fix: Restrict ONNX by setting sess_options.intra_op_num_threads = 1. This forces ONNX execution onto a single thread, keeping CPU cores free for FastAPI's event loop to stream packets instantly.

2. Persistent Connection Pooling (The TLS Trick)

   • Problem: Opening a new TLS/SSL connection to OpenAI per request adds 50–100ms latency.
   • The Fix: Maintain a persistent httpx.AsyncClient HTTP/2 connection pool on server startup. The proxy opens pre-warmed secure tunnels, routing requests instantly with zero TLS setup overhead.

3. Swap to orjson for Chunk Parsing

   • Problem: In an SSE stream, standard Python json.loads parses hundreds of delta chunks per second.
   • The Fix: Swap built-in json for orjson (written in Rust). It parses streaming LLM chunks up to 10x faster, dropping proxy overhead to near zero.

4. Cythonize the Sliding-Window Buffer

   • Problem: The sliding-window buffer performs frequent string slicing and bracket matching.
   • The Fix: Use Cython or mypyc to compile streaming.py directly into a C-extension binary module. Retains Python readability while executing string operations at native C speed.

---

🏢 Using LLM-Shield-Proxy in Production?

I am actively working with enterprise security teams to map out advanced compliance features. If your startup or organization is using LLM-Shield-Proxy to unblock LLM streaming or pass SOC 2/HIPAA audits, I would love to hear from you.

Email the core maintainer at [e-mail] to share your feedback, request a feature, or feature your team as a case study.