PromptGuard - LLM Output Validation Framework

A pip-installable guardrails library for validating LLM outputs. Enforces schema compliance, detects PII leaks, scores hallucination risk, filters toxic content, and validates structured outputs against Pydantic models. Works as middleware between any LLM provider and your application.

---

Why I Built This

Every production LLM application needs a safety net between the model and the user. I've seen teams ship LLM features where a single hallucinated SSN or a toxic sentence can cause real damage -- legal, reputational, or operational. Existing solutions either require paid API calls, pull in massive dependencies, or only solve one piece of the puzzle.

PromptGuard is my answer: a single, zero-cost library that chains together PII detection, toxicity scoring, hallucination risk analysis, and schema validation into one configurable pipeline. It runs entirely locally with no network calls, making it suitable for air-gapped environments and latency-sensitive applications. I built it to be the validation layer I wished I had on every LLM project.

---

Architecture

graph TD
    A[LLM Output Text] --> B[ValidationEngine]
    B --> C{Rule Chain from YAML Config}
    C --> D[PII Scanner]
    C --> E[Toxicity Scorer]
    C --> F[Hallucination Analyzer]
    C --> G[Schema Validator]
    D --> H[Regex Pattern Matching]
    D --> I[Luhn Check for CC]
    E --> J[Weighted Pattern Matching]
    E --> K[Category Aggregation]
    F --> L[Entity Extraction]
    F --> M[Claim Analysis]
    F --> N[Hedging Detection]
    G --> O[JSON Schema Validation]
    G --> P[Pydantic Model Validation]
    D --> Q[ValidationReport]
    E --> Q
    F --> Q
    G --> Q
    Q --> R{passed?}
    R -->|Yes| S[Return to Application]
    R -->|No| T[Block / Warn / Redact]

    style A fill:#e1f5fe
    style B fill:#f3e5f5
    style Q fill:#e8f5e9
    style T fill:#ffebee

Component Overview

Component	Purpose	Technique
PII Scanner	Detects emails, phones, SSNs, credit cards, IPs, DOBs	Compiled regex + Luhn algorithm
Toxicity Scorer	Scores harmful content from 0.0 to 1.0	Weighted keyword/pattern matching with category aggregation
Hallucination Analyzer	Estimates fabrication risk	Entity extraction, claim detection, hedging analysis, reference comparison
Schema Validator	Validates structured JSON output	JSON-schema-style rules or Pydantic model validation
Rule Engine	Chains validators into a pipeline	YAML-configurable with hard/soft fail modes

---

Quick Demo (60-Second Walkthrough)

Install

pip install -e .

Python API

from promptguard import ValidationEngine, scan_pii, score_toxicity, redact_pii

# Full pipeline validation
engine = ValidationEngine()
report = engine.validate("Contact john@example.com for the report.")
print(report.passed)    # False
print(report.summary)   # FAIL: 1 PII item(s) detected

# PII scanning
matches = scan_pii("SSN: 123-45-6789, Email: test@corp.com")
for m in matches:
    print(f"{m.pii_type.value}: {m.redacted}")
# ssn: ***-**-****
# email: t***@corp.com

# Redaction
safe = redact_pii("Call 555-123-4567 or email admin@company.com")
print(safe)  # Call ***-***-4567 or email a***@company.com

# Toxicity scoring
result = score_toxicity("This project is absolutely wonderful!")
print(result.score, result.is_toxic)  # 0.0 False

CLI

# Full validation
promptguard validate "The quarterly report shows 15% growth."

# Scan for PII
promptguard scan-pii "Email: john@example.com, SSN: 123-45-6789"

# Redact PII
promptguard scan-pii --redact "Contact: admin@corp.com"

# Quick CI/CD check (exit code 0=pass, 1=fail)
promptguard check "Clean text with no issues"

# Generate starter config
promptguard init-config -o promptguard.yml

# Use custom config
promptguard validate -c promptguard.yml "Some LLM output"

# JSON output for scripting
promptguard validate --json-output "Some text to validate"

YAML Configuration

version: "1"
name: my-pipeline
fail_fast: false
max_text_length: 100000
rules:
  - name: pii-scan
    validator: pii
    enabled: true
    hard_fail: true
    params: {}
  - name: toxicity-check
    validator: toxicity
    hard_fail: true
    params:
      threshold: 0.5
  - name: hallucination-risk
    validator: hallucination
    hard_fail: false  # warn but don't block
    params:
      threshold: 0.5
  - name: schema-check
    validator: schema
    hard_fail: true
    params:
      strict: true

---

Features

• Schema validation for JSON/structured LLM outputs (JSON-schema-style or Pydantic models)
• PII detection for emails, phones, SSNs, credit cards, IP addresses, dates of birth
• PII redaction with type-aware masking (e.g., j***@example.com, ***-**-****)
• Toxicity scoring with keyword and pattern matching across multiple categories
• Hallucination risk scoring via entity extraction, claim analysis, and hedging detection
• CLI tool with validate, scan-pii, and check commands
• Configurable rule chains with YAML config, hard/soft fail modes, fail-fast option
• Full type hints and py.typed marker for IDE support
• Zero external API calls -- runs entirely offline

---

Performance / Benchmarks

All benchmarks measured on a standard laptop (Intel i7, 16 GB RAM, Python 3.11).

Operation	Text Size	Time	Throughput
PII scan (all types)	1 KB	~0.2 ms	~5,000 texts/sec
PII scan (all types)	10 KB	~1.5 ms	~667 texts/sec
Toxicity scoring	1 KB	~0.1 ms	~10,000 texts/sec
Hallucination analysis	1 KB	~0.3 ms	~3,300 texts/sec
Full pipeline (3 rules)	1 KB	~0.8 ms	~1,250 texts/sec
Schema validation (Pydantic)	500 B	~0.05 ms	~20,000 texts/sec

Memory footprint: ~15 MB base (compiled regex patterns + Pydantic models). Scales linearly with text size during processing.

Key design decisions for performance:

• All regex patterns are pre-compiled at import time
• Luhn algorithm runs only on credit-card-shaped matches (not all digit sequences)
• Toxicity scoring uses short-circuit max-weight calculation
• The engine supports fail_fast mode to skip remaining rules on first failure

---

What I Would Do Differently

1. ML-based toxicity scoring: The keyword/pattern approach has clear limitations -- it misses subtle toxicity and can false-positive on clinical or academic text. A lightweight transformer model (e.g., a distilled BERT classifier) would dramatically improve accuracy while staying offline-capable.

2. Smarter hallucination detection: The current heuristic approach (hedging phrases, claim patterns, entity matching) is a starting point. Real hallucination detection needs access to the source documents and ideally the model's token-level probabilities. I'd integrate with retrieval pipelines to do proper attribution checking.

3. Streaming support: LLM outputs often stream token-by-token. The current API requires the complete text. A streaming validator that can flag issues incrementally (e.g., "PII detected at token 47") would be much more practical in production.

4. Plugin architecture: Instead of hardcoding four validators, I'd build a proper plugin system where custom validators can register via entry_points. This would let teams add domain-specific rules (e.g., medical term validation, financial compliance checks) without forking the library.

5. Async engine: The validation engine is synchronous. For high-throughput services, an async version with asyncio support would allow concurrent validation of multiple outputs.

---

Scaling Considerations

Horizontal Scaling

PromptGuard is stateless and thread-safe (all validators use compiled patterns and pure functions). You can run it across multiple processes or containers without coordination.

High-Throughput Pipelines

• Use fail_fast: true in config to short-circuit on the first failure
• Disable validators you don't need (e.g., skip hallucination for structured-output-only flows)
• For batch processing, the engine can be instantiated once and reused across thousands of validations

Memory Considerations

• Compiled regex patterns are shared across all engine instances in the same process
• For texts > 100KB, consider chunking and validating segments independently
• The max_text_length config parameter acts as a safety valve against OOM scenarios

Integration Patterns

[LLM Provider] --> [PromptGuard] --> [Your Application]
                        |
                   [Config YAML]

PromptGuard is designed to sit as middleware. Common integration points:

• FastAPI middleware: Validate every response before returning to the client
• LangChain callback: Plug into the chain as a post-processing step
• Batch pipeline: Validate CSV/JSONL outputs before downstream consumption

---

Development

# Clone and install in dev mode
git clone https://github.com/msoni029/promptguard.git
cd promptguard
pip install -e ".[dev]"

# Run tests
python -m pytest tests/ -v

# Type checking
mypy src/promptguard/

# Linting
ruff check src/ tests/

---

---

Sample Input / Output

---

Project Overview

Reports

• HTML Report - interactive report
• PDF Report - downloadable PDF
• TXT Report - plain text

License

MIT License. See LICENSE for details.