validate-llm 0.1.3

A composable validation framework for LLM inputs and outputs

Bring your own LLM — validate-llm wraps it with configurable input and output guardrails. Pass any provider and model (Anthropic, OpenAI, Gemini, and 100+ more), and get structured validation results for toxicity, privacy, accuracy, relevancy, and bias.

Documentation →

---

Install

pip install validate-llm

Requires Python 3.11+.

Quick start

from llm_validation_framework import ValidationFramework, LLMProvider, Pipe
from llm_validation_framework import ToxicityAgent, PrivacyAgent, AccuracyAgent
from llm_validation_framework.config_loader import load_api_key

llm = LLMProvider(provider="anthropic", model="claude-haiku-4-5-20251001", key=load_api_key())

vf = ValidationFramework(
    llm=llm,
    input_guardrail=Pipe(steps=[ToxicityAgent()], verbose=False),
    output_guardrail=Pipe(steps=[ToxicityAgent(), PrivacyAgent(), AccuracyAgent()], verbose=False),
)

result = vf.validate("What is the Pacific Ocean?")
print(result["status"], result["score"])  # PASS 0.87

validate() returns a structured dict with status, score, and per-agent results for both the input and output guardrails. See the docs for the full schema.

Demo

The demo is a FastAPI backend + static web UI.

# Terminal 1
uvicorn demo.api_server:app --host 127.0.0.1 --port 5050

# Terminal 2
python demo/serve_ui.py

Open http://127.0.0.1:8000.

Agents

Agent	What it does	Needs API key
ToxicityAgent	Three-layer check: profanity filter → toxicity model → semantic similarity	No
PrivacyAgent	Regex scan for SSN, credit cards, API keys; optional system prompt leakage detection	No
AccuracyAgent	LLM-as-a-judge factual accuracy + relevancy, with optional RAG grounding	Yes
RelevancyAgent	LLM-as-a-judge check that the answer addresses the question	Yes
BiasAgent	LLM-as-a-judge scan for stereotypes and discriminatory language	Yes

ToxicityAgent and PrivacyAgent run fully locally with no external calls.

Config

Set the API key for your chosen provider:

export ANTHROPIC_API_KEY=your-key   # Anthropic (default)
export OPENAI_API_KEY=your-key      # OpenAI
export GEMINI_API_KEY=your-key      # Google

Or create a config.ini at the repo root (gitignored):

[ANTHROPIC]
API_KEY=your-key

[OPENAI]
API_KEY=your-key

Any provider supported by litellm works. Pass provider= and model= to any LLM-as-a-judge agent to switch:

AccuracyAgent(provider="openai", model="gpt-4o-mini")
BiasAgent(provider="gemini", model="gemini-2.0-flash")

RAG grounding

Pass a retriever to AccuracyAgent to ground factual checks against your own corpus:

from llm_validation_framework import AccuracyAgent, RAGProvider

accuracy = AccuracyAgent(rag=RAGProvider(your_vectorstore.as_retriever()))

See the RAG Integration guide for a full walkthrough.

Contributors

• Hitha Shri Nagaruru
• James Wu
• Lewis Lui
• Thomas Yeoh

License

MIT — see LICENSE