LLM-Shield-Proxy

This repository contains two related packages:

1. pii-leak-benchmark tests an OpenAI-compatible streaming gateway. It checks whether the gateway sends the test values to its model provider and whether the client gets the original values back.
2. LLM-Shield-Proxy is a self-hosted streaming privacy gateway. The benchmark tests it by name and applies the same publication rules used for every other gateway.

What changed after the first benchmark run

The six results below were produced by this project on one workstation. No outside contributor has repeated them yet, so the table marks every product result as unreplicated. Each result links to the exact configuration and the report produced by the run.

The first version of the test used invalid examples of an email address, SSN, and credit card:

LLM-Shield-Proxy matched the text patterns but did not perform those validity checks. This gave it an unfair advantage over detectors that validate values. A LiteLLM and Presidio run revealed the problem: the old values produced leaked: ["SSN"], while valid test values produced leaked: []. The project did not publish the affected result. It replaced the three values with valid, reserved test values and reran all six configurations. See the fixture threat model for the full record.

The benchmark also found two streaming bugs in LLM-Shield-Proxy. It created an OpenTelemetry span for every SSE event even when export was off, and it sent each event's blank terminator as a separate write. Both bugs are fixed and covered by tests/test_streaming_write_efficiency.py. The run record explains what changed.

Known limitation: the test uses three fixed data formats. A small program written specifically for those formats can pass without being a general PII detector. The values change on every run, but the formats do not. Testing more formats caused two false failures in six trials. See the fixture threat model for the measurements.

Run it yourself, in about a minute

pip install pii-leak-benchmark

# The negative control: no gateway at all, raw pass-through. MUST report outcome=fail.
pii-leak-benchmark \
  --target-base-url capture://self \
  --target-name raw-pass-through-negative-control --target-version 1 \
  --redaction-claimed claimed \
  --redaction-claim-citation https://github.com/ninadphalak/LLM-Shield-Proxy/blob/main/website/docs/conformance/reproducing.md \
  --redaction-enabled \
  --redaction-config-reference "synthetic control: declared redaction intentionally absent"

# Your gateway, already configured to send upstream traffic to http://127.0.0.1:8765/v1
pii-leak-benchmark --target-base-url http://127.0.0.1:4000/v1 --target-name your-gateway

Reproduce a published result instead of measuring a gateway

One bounded experiment, offline, no gateway or account. It re-runs the chunk-local and length-bounded-retention inspectors at the published seed and diffs every field of the result against the published reports:

git clone https://github.com/ninadphalak/LLM-Shield-Proxy.git
cd LLM-Shield-Proxy
python -m pip install ./pii-leak-benchmark
python benchmarks/reproduce_fragmentation.py --out reproduction

About two minutes. Exit status 0 means every field matched except timestamps and wall-clock timings. The same command runs in CI on Ubuntu, macOS and Windows across Python 3.11 and 3.12 (fragmentation-reproduction in .github/workflows/benchmark.yml). Full walkthrough and what the numbers mean: reproduce the fragmentation result.

Check your own gateway

One question, one command: does your deployment send raw personal data to its upstream?

pip install "pii-leak-benchmark>=0.2.0"

# Establish the floor first. No gateway at all: this MUST report LEAK.
pii-leak-benchmark selfcheck --target-base-url capture://self

# Then your own gateway, already configured to use the capture as its upstream.
pii-leak-benchmark selfcheck --target-base-url http://your-gateway.internal/v1

selfcheck needs 0.2.0 or newer.

To fail your own build on a leak, the complete workflow file and the three settings you have to change are in the CI setup guide, with a shorter copy in examples/ci/gateway-pii-check.yml. That path needs 0.3.0 or newer, which adds the ci subcommand, the GitHub Action, and baseline comparison against your previous version.

It reports one row per data type, so you can see what your gateway handled and what it missed:

    TYPE               RESULT        WHAT IT MEANS
    AWS_ACCESS_KEY_ID  LEAK          sent to the upstream unmasked
    CREDIT_CARD        contained     never reached the upstream in this run
    EMAIL              LEAK          sent to the upstream unmasked
    GITHUB_TOKEN       LEAK          sent to the upstream unmasked
    SLACK_TOKEN        LEAK          sent to the upstream unmasked
    SSN                contained     never reached the upstream in this run

That is a real result from a gateway that redacts SSNs and card numbers and forwards every credential, which is the common shape. Six types: email, SSN, card, plus AWS, GitHub and Slack credential specimens. Every run also prints what it did not test, because the dangerous reading of a clean result is "my gateway handles sensitive data".

Exit 0 CLEAN, 1 LEAK, 2 NOT MEASURED. The third is the one that matters: a gateway that answers your client but was never pointed at the capture inspects nothing, so every check passes vacuously. That gets its own exit code instead of reading as a pass. selfcheck requires no vendor claim and its report is deliberately not publishable as a row about a product; to publish a comparative result, use the flat command and record the claim.

The only third-party Python dependency is httpx; you do not need to install one gateway to test another. You configure the gateway to use the benchmark's local capture server as its model provider. The benchmark then checks the URL, headers, HTTP framing, and JSON body for the test values. If it cannot safely parse part of the request, the run ends with an error instead of assuming that no value leaked. The conformance docs describe the full method.

fail has one narrow meaning: the gateway sent an unmasked test value to the benchmark's capture server. A product that does not offer PII redaction is marked not-applicable, not failed. A one-way anonymizer that removes the values but does not restore them receives a separate outcome.

Results

Every product result above was run once by this project's maintainer. It has not yet been repeated by an independent person. A result becomes replicated only after three different people each submit a run of the same gateway and configuration. Until then, it remains unreplicated. Full table, method and evidence · submit a run.

Run LLM-Shield-Proxy

pip install llm-shield-proxy
llm-shield-proxy --host 0.0.0.0 --port 8000
curl http://localhost:8000/healthz

For the container path:

docker compose up -d
curl http://localhost:8000/healthz
python examples/demo.py

LLM-Shield-Proxy is a self-hosted privacy gateway for OpenAI-compatible streaming APIs. It applies configured PII, PHI, PCI and secret transformations before the upstream, then rehydrates the masked values incrementally as SSE events arrive. Point an existing client at it by changing base_url:

from openai import OpenAI

client = OpenAI(api_key="your-shield-virtual-key", base_url="http://localhost:8000/v1")
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Contact Sarah at sarah@example.com."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

A successful health check only means the server started. To send a model request, add the API key for your model provider and configure a key that clients will use to call the proxy. Start with .env.example, then follow the deployment guide.

How LLM-Shield-Proxy works

Before sending a request to the model provider, the proxy finds configured types of sensitive data and replaces their values. As the provider streams its response, the proxy joins replacement tokens that were split across SSE events and restores values that the client is allowed to receive. For structured JSON, it changes string values without changing the JSON syntax. Test this behavior with the schemas used by your provider and tools.

The maintained component map and deployment diagrams live in the architecture guide, architecture whitepaper, and deployment guide.

Deployment choices

In standard mode, detection, masking, policy checks, and value restoration run inside your gateway. Only the masked request is sent to the external model provider:

In air-gapped mode, the masked request goes to an internal model gateway. Network policy must still block direct provider access, telemetry, and other unintended outbound traffic:

See deployment topologies, air-gapped egress, and the Kubernetes/Helm deployment guide.

Every feature is labelled Supported, Beta, Experimental, or Research. The label states how the feature was tested and what remains untested: feature catalog · stability policy · limitations.

It supports SOC 2, HIPAA, GDPR, EU AI Act and NIST/ISO evidence programs by supplying technical controls and artifacts. It does not certify a deployment, guarantee complete detection, or make network policy optional.

Verifying this repository

git clone https://github.com/ninadphalak/LLM-Shield-Proxy.git
cd LLM-Shield-Proxy
python -m pip install -e ./pii-leak-benchmark -e ".[dev]"
python -m pytest

The benchmark is a separate distribution in this repo, so it installs first; nothing in it imports the proxy and a test fails if that ever changes. CI provisions real Redis, an HTTP/2 ALPN server, a checksum-pinned ONNX export, Docker, Helm and promtool. A missing dependency fails those jobs rather than skipping them, so a green build cannot mean "nothing ran".

Documentation

• Start here: interactive docs and playground · configuration · deployment · operations · troubleshooting
• Understand the design: architecture · architecture whitepaper · feature catalog · stability · limitations
• Integrate it: integration index · LiteLLM and Ollama recipe · Open WebUI and LangChain recipe · migration from Presidio
• Operate securely: security model · policy as code · compliance evidence mapping · immutable retention
• Verify and participate: conformance method and results · submit a reproduction · 30-day design-partner pilot

Contributing, license, and citation

Contributions are welcome through issues, discussions and CONTRIBUTING.md. The most valuable contribution is an independent benchmark run against a gateway you operate, whether it matches or differs from a row above.

Source code is Apache 2.0; documentation and diagrams may carry CC BY 4.0 terms. See LICENSE.

The author identifies U.S. application numbers 64/126,730 and 64/139,263 as pending filings related to streaming transformation and structured stateless masking. Pending applications are not issued patents; verify status with counsel and official records before relying on them.

If you reference the architecture or benchmark methodology, use CITATION.cff or:

Phalak, N. (2026). Quantifying Latency and Token Overhead in Real-Time LLM Stream Sanitization: A Tiered Detection Approach. https://doi.org/10.5281/zenodo.21955770