VeilLLM

A local DLP reverse proxy for LLM traffic. Mask PII, API keys, and secrets before they leave your machine — restore them on the way back.

https://github.com/user-attachments/assets/dec6fb5d-64d2-4731-a4b3-40260ff8f461

Quickstart

# Install (pipx recommended — isolated env, global CLI)
pipx install veil-llm

# Start the gateway
veil-llm serve --port 4000

# Show resolved settings
veil-llm config

pip install veil-llm also works if you don't use pipx.

On first run, VeilLLM creates a ./config.toml with a random salt and default settings. The spaCy model (en_core_web_lg, ~500MB) downloads automatically if missing.

CLI reference

veil-llm serve [options]
veil-llm config            # show resolved settings
veil-llm --help            # full help

Flag	Env var	Default	Description
--port	GATEWAY_PORT	4000	Listen port
--host	—	127.0.0.1	Listen host
--upstream-url	GATEWAY_UPSTREAM_URL	https://openrouter.ai/api/v1	Upstream API base URL
--model-name	GATEWAY_MODEL_NAME	en_core_web_lg	spaCy model for PII detection
--debug	GATEWAY_DEBUG=1	false	Log placeholder→original map to stderr (⚠️ leaks secrets)
--high-entropy	GATEWAY_HIGH_ENTROPY=1	false	Enable high-entropy string detection
--entropy-threshold	GATEWAY_ENTROPY_THRESHOLD	4.5	Shannon entropy threshold for high-entropy detection
--enabled-entities	GATEWAY_ENABLED_ENTITIES	PERSON,EMAIL_ADDRESS,…	Comma-separated list of PII entity types to detect

Priority: config.toml defaults < config.toml values < env vars < CLI flags.

Available spaCy models

Model	Size	Vectors	Accuracy	Best for
en_core_web_sm	12 MB	❌	Low	Minimal footprint, quick tests
en_core_web_md	40 MB	✅	Medium	Balanced — smaller than lg, better NER than sm
en_core_web_lg	560 MB	✅	High	Best CPU accuracy (default)
en_core_web_trf	460 MB	✅	Highest	Transformer-based; slowest, needs GPU for practical use

Use --model-name or config to switch:

veil-llm serve --model-name en_core_web_md

Usage with any agentic harness

VeilLLM speaks the OpenAI-compatible API. Any harness that can point to a custom base URL — Hermes, Cline, Continue, Aider, OpenHands, or your own scripts — works with one config change:

# Before: harness hits provider directly
base_url: https://api.openai.com/v1

# After: harness hits VeilLLM, which masks then forwards
base_url: http://127.0.0.1:4000/v1

Your Authorization header is forwarded through unchanged. VeilLLM masks sensitive data in your prompts and restores it in responses transparently — the harness never knows it's there.

Important: VeilLLM v0.1 does not support streaming (stream: true). Configure your harness for non-streaming mode.

Configuration

VeilLLM auto-creates ./config.toml on first run with sensible defaults:

[gateway]
upstream_url = "https://openrouter.ai/api/v1"
port = 4000
salt = "<auto-generated>"
model_name = "en_core_web_lg"

[detection]
enabled_entities = [
    "PERSON",
    "EMAIL_ADDRESS",
    "PHONE_NUMBER",
    "CREDIT_CARD",
    "IBAN_CODE",
    "IP_ADDRESS",
    "LOCATION",
    "US_SSN",
]

[recognizers]
high_entropy_enabled = false
entropy_threshold = 4.5

Every setting can be overridden via environment variable or CLI flag (see table above). Run veil-llm config to see the resolved settings your gateway will use.

What gets masked

Category	Examples
PII (Presidio)	Names, emails, phone numbers, credit cards, IBANs, IPs, locations, SSNs
LLM API keys	sk-..., sk-ant-..., sk-or-...
AWS keys	AKIA... access keys and secret keys
GitHub tokens	ghp_..., gho_..., github_pat_...
Google API keys	AIza...
JWTs	eyJ... three-segment tokens
Private keys	PEM -----BEGIN ... PRIVATE KEY----- blocks
High-entropy strings	Opt-in: strings with Shannon entropy > threshold

Gradio playground

For a hands-on feel, launch the Gradio UI to see masking live — no upstream API key needed for the detection tab:

uv run --with gradio --with python-dotenv python playground.py

Open http://127.0.0.1:7860. The PII Detection tab lets you paste text and see exactly what gets masked. The Chat tab sends masked prompts to an LLM and shows a transparency log of every entity detected, masked, and restored — the same thing the proxy does silently.

playground.py is a self-contained demo; read it alongside this README to understand the full flow.

How it works

VeilLLM replaces detected values with deterministic placeholders:

"Jane" → <PERSON_a3f2>
"jane@example.com" → <EMAIL_9c1d4e>
"sk-abc123..." → <API_KEY_77b0af>

The same value always maps to the same placeholder (salted hash), so agent conversation history stays coherent across turns. The salt is a random value in ./config.toml — never sent over the network.

On the response, placeholders are restored using tolerant regex matching (handles LLM-mangled casing, spaces, and punctuation).

Endpoints

Endpoint	Behavior
POST /v1/chat/completions	Mask → forward → unmask
GET /v1/models	Pass-through
GET /healthz	Liveness check
Any other /v1/*	Pass-through (no masking, warning logged)

Telemetry

Per-request log line (stderr):

[req a3f2bc8d] masked 3 entities: API_KEY×1, EMAIL_ADDRESS×1, PERSON×1 | detect 35ms | upstream 1850ms | unmask 2ms

Set GATEWAY_DEBUG=1 to see the placeholder→original map (with a prominent warning).

Docker

docker build -t veilllm .
docker run -p 4000:4000 -v ./config.toml:/app/config.toml veilllm

Limitations (v0.1)

• No streaming (SSE) — returns 400 if stream: true
• English-only PII detection
• Single upstream URL (defaults to OpenRouter; any OpenAI-compatible API works via GATEWAY_UPSTREAM_URL or config.toml)
• Not a full proxy — only the /v1/* paths are forwarded

License

MIT