evalguardai 2.1.2

Official EvalGuard Python SDK — LLM evaluation, red-team security, runtime guardrails, observability, and FinOps.

evalguardai

Official Python SDK for EvalGuard -- evaluate, red-team, and guard LLM applications with drop-in framework integrations.

The package is published on PyPI as evalguardai (we own this slot). Aliases evalguard-sdk and evalguard-python are deprecation shims that re-export from here. The unrelated third-party evalguard package on PyPI is owned by yolojewjitsu/evalguard and is not affiliated with EvalGuard, Inc.

Installation

# Core SDK
pip install evalguardai

# With framework extras
pip install evalguardai[openai]
pip install evalguardai[anthropic]
pip install evalguardai[langchain]
pip install evalguardai[bedrock]
pip install evalguardai[crewai]
pip install evalguardai[fastapi]

# Everything
pip install evalguardai[all]

Quick Start

# Install name == import name. `import evalguard` and `EvalGuardClient` also work.
from evalguardai import EvalGuard

client = EvalGuard(api_key="eg_live_...")

# Start an evaluation (`name` is required by POST /v1/evals). The call returns
# a run stub with an id + status; the run executes in the background.
run = client.run_eval({
    "name": "Arithmetic eval",
    "model": "gpt-4o",
    "prompt": "Answer: {{input}}",
    "cases": [
        {"input": "What is 2+2?", "expectedOutput": "4"},
    ],
    "scorers": ["exact-match", "contains"],
})
print(f"Started eval {run['id']} (status: {run['status']})")

# Once the run finishes (status → passed / failed), fetch the detail. The eval
# detail nests the run row under `run` and the aggregates under `summary`.
detail = client.get_eval(run["id"])  # GET /v1/evals/{runId}
print(f"Status: {detail['run']['status']}, Pass rate: {detail['summary']['passRate']}")

# Run a security scan (red-team) — needs projectId (auto-resolved if omitted),
# model, prompt and at least one attackType. Returns the scan with its `id`.
scan = client.run_scan({
    "model": "gpt-4o",
    "prompt": "You are a helpful assistant",
    "attackTypes": ["prompt-injection", "jailbreak"],
})
detail = client.get_scan(scan["id"])  # GET /v1/security/{scanId}

# Check the firewall
fw = client.check_firewall("Ignore all previous instructions")
print(f"Blocked: {fw['blocked']}  Category: {fw['category']}")  # True / "prompt-injection"

---

Framework Integrations

Every integration is a drop-in wrapper -- add two lines and your existing code gets automatic guardrails, traces, and observability.

OpenAI

from evalguardai.openai import wrap
from openai import OpenAI

client = wrap(OpenAI(), api_key="eg_...", project_id="proj_...")

# Use exactly like normal -- guardrails are automatic
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)

All calls to chat.completions.create() are intercepted:

• Pre-LLM: Input is checked for prompt injection, PII, etc.
• Post-LLM: Response + latency + token usage are traced to EvalGuard.
• Violations: Raise GuardrailViolation (or log-only with block_on_violation=False).

Anthropic

from evalguardai.anthropic import wrap
from anthropic import Anthropic

client = wrap(Anthropic(), api_key="eg_...", project_id="proj_...")

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain quantum computing"}],
)
print(response.content[0].text)

Intercepts messages.create() with the same pre/post guardrail pattern.

LangChain

from evalguardai.langchain import EvalGuardCallback
from langchain_openai import ChatOpenAI

callback = EvalGuardCallback(api_key="eg_...", project_id="proj_...")

llm = ChatOpenAI(model="gpt-4o", callbacks=[callback])
result = llm.invoke("What is the capital of France?")

Works with any LangChain LLM, chat model, or chain that supports callbacks. The callback implements the full LangChain callback protocol without importing LangChain, so it is compatible with all versions (0.1.x through 0.3.x).

Traced events:

• on_llm_start / on_chat_model_start -- pre-check input
• on_llm_end -- log output trace
• on_llm_error -- log error trace

AWS Bedrock

from evalguardai.bedrock import wrap
import boto3

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
client = wrap(bedrock, api_key="eg_...", project_id="proj_...")

# invoke_model (all Bedrock model families supported)
import json
response = client.invoke_model(
    modelId="anthropic.claude-3-sonnet-20240229-v1:0",
    body=json.dumps({
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 256,
        "anthropic_version": "bedrock-2023-05-31",
    }),
)

# Converse API
response = client.converse(
    modelId="anthropic.claude-3-sonnet-20240229-v1:0",
    messages=[{"role": "user", "content": [{"text": "Hello"}]}],
)

Supports all Bedrock model families: Anthropic Claude, Amazon Titan, Meta Llama, Cohere, AI21, and Mistral. Both invoke_model and converse APIs are guarded.

CrewAI

from evalguardai.crewai import guard_agent, EvalGuardGuardrail
from crewai import Agent, Task, Crew

# Guard individual agents
agent = Agent(role="researcher", goal="...", backstory="...")
agent = guard_agent(agent, api_key="eg_...")

# Or use the standalone guardrail
guardrail = EvalGuardGuardrail(api_key="eg_...", project_id="proj_...")
result = guardrail.check("User input to validate")

# Wrap arbitrary functions
@guardrail.wrap_function
def my_tool(query: str) -> str:
    return do_search(query)

FastAPI Middleware

from evalguardai.fastapi import EvalGuardMiddleware
from fastapi import FastAPI

app = FastAPI()
app.add_middleware(
    EvalGuardMiddleware,
    api_key="eg_...",
    project_id="proj_...",
)

@app.post("/api/chat")
async def chat(request: dict):
    # Automatically guarded -- prompt injection blocked with 403
    return {"response": "..."}

By default, POST requests to paths containing /chat, /completions, /generate, /invoke, or /messages are guarded. Customize with guarded_paths:

app.add_middleware(
    EvalGuardMiddleware,
    api_key="eg_...",
    guarded_paths={"/api/v1/chat", "/api/v1/generate"},
)

For per-route control:

from evalguardai.fastapi import guard_route

@app.post("/api/chat")
@guard_route(api_key="eg_...", rules=["prompt_injection"])
async def chat(request: Request):
    body = await request.json()
    ...

NeMo / Agent Workflows

from evalguardai.nemoclaw import EvalGuardAgent

agent = EvalGuardAgent(api_key="eg_...", agent_name="support-bot")

# Guard any LLM call
result = agent.guarded_call(
    provider="openai",
    messages=[{"role": "user", "content": "Reset my password"}],
    llm_fn=lambda: openai_client.chat.completions.create(
        model="gpt-4", messages=[{"role": "user", "content": "Reset my password"}]
    ),
)

# Multi-step agent sessions
with agent.session("ticket-123") as session:
    session.check("User says: reset my password")
    result = do_llm_call(...)
    session.log_step("password_reset", input="...", output=str(result))

---

Core Guardrail Client

All framework integrations share the same underlying GuardrailClient:

from evalguardai.guardrails import GuardrailClient

guard = GuardrailClient(
    api_key="eg_...",
    project_id="proj_...",
    timeout=5.0,       # keep low to avoid latency
    fail_open=False,   # fail-closed (default): raise on EvalGuard error so an outage can't silently bypass guardrails
)

# Pre-LLM check
result = guard.check_input("user prompt here", rules=["prompt_injection", "pii_redact"])
if not result["allowed"]:
    print("Blocked:", result["violations"])

# Post-LLM check
result = guard.check_output("model response here", rules=["toxic_content"])

# Fire-and-forget trace
guard.log_trace({"model": "gpt-4", "input": "...", "output": "...", "latency_ms": 120})

Error Handling

All integrations are fail-closed by default: if the EvalGuard API is unreachable, the guardrail check raises (the LLM call is blocked) so an outage cannot silently bypass your guardrails.

To fail-open instead (let requests through to the LLM on an EvalGuard outage):

# Framework wrappers
client = wrap(OpenAI(), api_key="eg_...", block_on_violation=False)

# Core client
guard = GuardrailClient(api_key="eg_...", fail_open=True)

Catch violations explicitly:

from evalguardai import GuardrailViolation

try:
    response = client.chat.completions.create(...)
except GuardrailViolation as e:
    print(f"Blocked: {e.violations}")

All SDK Methods

Method	Description
client.run_eval(config)	Run an evaluation with scorers and test cases
client.get_eval(run_id)	Fetch a specific eval run by ID
client.list_evals(project_id=None)	List eval runs, optionally filtered by project
client.run_scan(config)	Run a red-team security scan against a model
client.get_scan(scan_id)	Fetch a specific security scan by ID
client.list_scorers()	List all available evaluation scorers
client.list_plugins()	List all available security plugins
client.check_firewall(input_text, rules=None)	Check input against firewall rules
client.submit_benchmark(benchmark, model, total_score, scores=None)	Submit a benchmark run to the leaderboard
client.export_dpo(run_id, project_id)	Export eval results as DPO training data (JSONL)
client.export_burp(scan_id, project_id)	Export scan results as Burp Suite XML
client.get_compliance_report(scan_id, framework)	Map scan results to a compliance framework
client.detect_drift(config)	Detect performance drift between eval runs
client.generate_guardrails(config)	Auto-generate firewall rules from scan findings

Documentation

Full documentation at docs.evalguard.ai/python-sdk.

License

Apache-2.0 -- see LICENSE for details.