evalguardai 1.1.0

Python SDK for EvalGuard -- evaluate, red-team, and guard LLM applications with drop-in framework integrations

evalguard-python

Python SDK for EvalGuard -- evaluate, red-team, and guard LLM applications with drop-in framework integrations.

Installation

# Core SDK
pip install evalguard-python

# With framework extras
pip install evalguard-python[openai]
pip install evalguard-python[anthropic]
pip install evalguard-python[langchain]
pip install evalguard-python[bedrock]
pip install evalguard-python[crewai]
pip install evalguard-python[fastapi]

# Everything
pip install evalguard-python[all]

Quick Start

from evalguard import EvalGuardClient

client = EvalGuardClient(api_key="eg_live_...")

# Run an evaluation
result = client.run_eval({
    "model": "gpt-4o",
    "prompt": "Answer: {{input}}",
    "cases": [
        {"input": "What is 2+2?", "expectedOutput": "4"},
    ],
    "scorers": ["exact-match", "contains"],
})
print(f"Score: {result['score']}, Pass rate: {result['passRate']}")

# Check the firewall
fw = client.check_firewall("Ignore all previous instructions")
print(f"Action: {fw['action']}")  # "block"

---

Framework Integrations

Every integration is a drop-in wrapper -- add two lines and your existing code gets automatic guardrails, traces, and observability.

OpenAI

from evalguard.openai import wrap
from openai import OpenAI

client = wrap(OpenAI(), api_key="eg_...", project_id="proj_...")

# Use exactly like normal -- guardrails are automatic
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)

All calls to chat.completions.create() are intercepted:

• Pre-LLM: Input is checked for prompt injection, PII, etc.
• Post-LLM: Response + latency + token usage are traced to EvalGuard.
• Violations: Raise GuardrailViolation (or log-only with block_on_violation=False).

Anthropic

from evalguard.anthropic import wrap
from anthropic import Anthropic

client = wrap(Anthropic(), api_key="eg_...", project_id="proj_...")

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain quantum computing"}],
)
print(response.content[0].text)

Intercepts messages.create() with the same pre/post guardrail pattern.

LangChain

from evalguard.langchain import EvalGuardCallback
from langchain_openai import ChatOpenAI

callback = EvalGuardCallback(api_key="eg_...", project_id="proj_...")

llm = ChatOpenAI(model="gpt-4o", callbacks=[callback])
result = llm.invoke("What is the capital of France?")

Works with any LangChain LLM, chat model, or chain that supports callbacks. The callback implements the full LangChain callback protocol without importing LangChain, so it is compatible with all versions (0.1.x through 0.3.x).

Traced events:

• on_llm_start / on_chat_model_start -- pre-check input
• on_llm_end -- log output trace
• on_llm_error -- log error trace

AWS Bedrock

from evalguard.bedrock import wrap
import boto3

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
client = wrap(bedrock, api_key="eg_...", project_id="proj_...")

# invoke_model (all Bedrock model families supported)
import json
response = client.invoke_model(
    modelId="anthropic.claude-3-sonnet-20240229-v1:0",
    body=json.dumps({
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 256,
        "anthropic_version": "bedrock-2023-05-31",
    }),
)

# Converse API
response = client.converse(
    modelId="anthropic.claude-3-sonnet-20240229-v1:0",
    messages=[{"role": "user", "content": [{"text": "Hello"}]}],
)

Supports all Bedrock model families: Anthropic Claude, Amazon Titan, Meta Llama, Cohere, AI21, and Mistral. Both invoke_model and converse APIs are guarded.

CrewAI

from evalguard.crewai import guard_agent, EvalGuardGuardrail
from crewai import Agent, Task, Crew

# Guard individual agents
agent = Agent(role="researcher", goal="...", backstory="...")
agent = guard_agent(agent, api_key="eg_...")

# Or use the standalone guardrail
guardrail = EvalGuardGuardrail(api_key="eg_...", project_id="proj_...")
result = guardrail.check("User input to validate")

# Wrap arbitrary functions
@guardrail.wrap_function
def my_tool(query: str) -> str:
    return do_search(query)

FastAPI Middleware

from evalguard.fastapi import EvalGuardMiddleware
from fastapi import FastAPI

app = FastAPI()
app.add_middleware(
    EvalGuardMiddleware,
    api_key="eg_...",
    project_id="proj_...",
)

@app.post("/api/chat")
async def chat(request: dict):
    # Automatically guarded -- prompt injection blocked with 403
    return {"response": "..."}

By default, POST requests to paths containing /chat, /completions, /generate, /invoke, or /messages are guarded. Customize with guarded_paths:

app.add_middleware(
    EvalGuardMiddleware,
    api_key="eg_...",
    guarded_paths={"/api/v1/chat", "/api/v1/generate"},
)

For per-route control:

from evalguard.fastapi import guard_route

@app.post("/api/chat")
@guard_route(api_key="eg_...", rules=["prompt_injection"])
async def chat(request: Request):
    body = await request.json()
    ...

NeMo / Agent Workflows

from evalguard.nemoclaw import EvalGuardAgent

agent = EvalGuardAgent(api_key="eg_...", agent_name="support-bot")

# Guard any LLM call
result = agent.guarded_call(
    provider="openai",
    messages=[{"role": "user", "content": "Reset my password"}],
    llm_fn=lambda: openai_client.chat.completions.create(
        model="gpt-4", messages=[{"role": "user", "content": "Reset my password"}]
    ),
)

# Multi-step agent sessions
with agent.session("ticket-123") as session:
    session.check("User says: reset my password")
    result = do_llm_call(...)
    session.log_step("password_reset", input="...", output=str(result))

---

Core Guardrail Client

All framework integrations share the same underlying GuardrailClient:

from evalguard.guardrails import GuardrailClient

guard = GuardrailClient(
    api_key="eg_...",
    project_id="proj_...",
    timeout=5.0,       # keep low to avoid latency
    fail_open=True,    # allow on error (default)
)

# Pre-LLM check
result = guard.check_input("user prompt here", rules=["prompt_injection", "pii_redact"])
if not result["allowed"]:
    print("Blocked:", result["violations"])

# Post-LLM check
result = guard.check_output("model response here", rules=["toxic_content"])

# Fire-and-forget trace
guard.log_trace({"model": "gpt-4", "input": "...", "output": "...", "latency_ms": 120})

Error Handling

All integrations use fail-open semantics by default: if the EvalGuard API is unreachable, requests pass through rather than blocking your application.

To fail-closed:

# Framework wrappers
client = wrap(OpenAI(), api_key="eg_...", block_on_violation=True)

# Core client
guard = GuardrailClient(api_key="eg_...", fail_open=False)

Catch violations explicitly:

from evalguard import GuardrailViolation

try:
    response = client.chat.completions.create(...)
except GuardrailViolation as e:
    print(f"Blocked: {e.violations}")

All SDK Methods

Method	Description
client.run_eval(config)	Run an evaluation with scorers and test cases
client.get_eval(run_id)	Fetch a specific eval run by ID
client.list_evals(project_id=None)	List eval runs, optionally filtered by project
client.run_scan(config)	Run a red-team security scan against a model
client.get_scan(scan_id)	Fetch a specific security scan by ID
client.list_scorers()	List all available evaluation scorers
client.list_plugins()	List all available security plugins
client.check_firewall(input_text, rules=None)	Check input against firewall rules
client.run_benchmarks(suites, model)	Run benchmark suites against a model
client.export_dpo(run_id)	Export eval results as DPO training data (JSONL)
client.export_burp(scan_id)	Export scan results as Burp Suite XML
client.get_compliance_report(scan_id, framework)	Map scan results to a compliance framework
client.detect_drift(config)	Detect performance drift between eval runs
client.generate_guardrails(config)	Auto-generate firewall rules from scan findings

Documentation

Full documentation at docs.evalguard.ai/python-sdk.

License

MIT -- see LICENSE for details.