evalguardai

Official Python SDK for EvalGuard -- evaluate, red-team, and guard LLM applications with drop-in framework integrations.

The package is published on PyPI as evalguardai (we own this slot). Aliases evalguard-sdk and evalguard-python are deprecation shims that re-export from here. The unrelated third-party evalguard package on PyPI is owned by yolojewjitsu/evalguard and is not affiliated with EvalGuard, Inc.

:warning: pip install evalguard BREAKS this SDK — use pip install evalguardai

The bare evalguard distribution on PyPI is an unaffiliated third party's package (v0.1.0, a check / expect API) that claims the same top-level evalguard import name this SDK installs. pip installs one over the other with no warning, and both import names then stop working. Measured in a clean venv:

$ pip install evalguardai && pip install evalguard      # pip: no warning, exit 0
$ pip list | grep evalguard
evalguard          0.1.0
evalguardai        2.2.1
$ python -c "import evalguard; print(evalguard.__version__)"
0.1.0                              # theirs; no EvalGuardClient
$ python -c "from evalguardai import EvalGuard"
ImportError: cannot import name 'EvalGuard' from 'evalguard'

Note the last error names evalguard — a module you never typed. Since 2.2.2 the SDK detects this at import time and raises EvalGuardNameCollisionError (an ImportError subclass) naming the real cause and the fix:

pip uninstall -y evalguard evalguardai && pip install evalguardai

If you genuinely need that other package, install it in a separate environment.

Installation

# Core SDK — NOTE: `evalguardai`, not `evalguard` (see the warning above)
pip install evalguardai

# With framework extras
pip install evalguardai[openai]
pip install evalguardai[anthropic]
pip install evalguardai[langchain]
pip install evalguardai[bedrock]
pip install evalguardai[crewai]
pip install evalguardai[fastapi]

# Everything
pip install evalguardai[all]

Quick Start

# Install name == import name. `import evalguard` and `EvalGuardClient` also work.
from evalguardai import EvalGuard

client = EvalGuard(api_key="eg_live_...")

# Start an evaluation (`name` is required by POST /v1/evals). The call returns
# a run stub with an id + status; the run executes in the background.
run = client.run_eval({
    "name": "Arithmetic eval",
    "model": "gpt-4o",
    "prompt": "Answer: {{input}}",
    "cases": [
        {"input": "What is 2+2?", "expectedOutput": "4"},
    ],
    "scorers": ["exact-match", "contains"],
})
print(f"Started eval {run['id']} (status: {run['status']})")

# Once the run finishes (status → passed / failed), fetch the detail. The eval
# detail nests the run row under `run` and the aggregates under `summary`.
detail = client.get_eval(run["id"])  # GET /v1/evals/{runId}
print(f"Status: {detail['run']['status']}, Pass rate: {detail['summary']['passRate']}")

# Run a security scan (red-team) — needs projectId (auto-resolved if omitted),
# model, prompt and at least one attackType. Returns the scan with its `id`.
scan = client.run_scan({
    "model": "gpt-4o",
    "prompt": "You are a helpful assistant",
    "attackTypes": ["prompt-injection", "jailbreak"],
})
detail = client.get_scan(scan["id"])  # GET /v1/security/{scanId}

# Check the firewall
fw = client.check_firewall("Ignore all previous instructions")
print(f"Blocked: {fw['blocked']}  Category: {fw['category']}")  # True / "prompt-injection"

Framework Integrations

Every integration is a drop-in wrapper -- add two lines and your existing code gets automatic guardrails, traces, and observability.

OpenAI

from evalguardai.openai import wrap
from openai import OpenAI

client = wrap(OpenAI(), api_key="eg_...", project_id="proj_...")

# Use exactly like normal -- guardrails are automatic
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)

All calls to chat.completions.create() are intercepted:

• Pre-LLM: Input is checked for prompt injection, PII, etc.
• Post-LLM: Response + latency + token usage are traced to EvalGuard.
• Violations: Raise GuardrailViolation (or log-only with block_on_violation=False).

Anthropic

from evalguardai.anthropic import wrap
from anthropic import Anthropic

client = wrap(Anthropic(), api_key="eg_...", project_id="proj_...")

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain quantum computing"}],
)
print(response.content[0].text)

Intercepts messages.create() with the same pre/post guardrail pattern.

LangChain

from evalguardai.langchain import EvalGuardCallback
from langchain_openai import ChatOpenAI

callback = EvalGuardCallback(api_key="eg_...", project_id="proj_...")

llm = ChatOpenAI(model="gpt-4o", callbacks=[callback])
result = llm.invoke("What is the capital of France?")

Works with any LangChain LLM, chat model, or chain that supports callbacks. The callback implements the full LangChain callback protocol without importing LangChain, so it is compatible with all versions (0.1.x through 0.3.x).

Traced events:

• on_llm_start / on_chat_model_start -- pre-check input
• on_llm_end -- log output trace
• on_llm_error -- log error trace

AWS Bedrock

from evalguardai.bedrock import wrap
import boto3

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
client = wrap(bedrock, api_key="eg_...", project_id="proj_...")

# invoke_model (all Bedrock model families supported)
import json
response = client.invoke_model(
    modelId="anthropic.claude-3-sonnet-20240229-v1:0",
    body=json.dumps({
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 256,
        "anthropic_version": "bedrock-2023-05-31",
    }),
)

# Converse API
response = client.converse(
    modelId="anthropic.claude-3-sonnet-20240229-v1:0",
    messages=[{"role": "user", "content": [{"text": "Hello"}]}],
)

Supports all Bedrock model families: Anthropic Claude, Amazon Titan, Meta Llama, Cohere, AI21, and Mistral. Both invoke_model and converse APIs are guarded.

CrewAI

from evalguardai.crewai import guard_agent, EvalGuardGuardrail
from crewai import Agent, Task, Crew

# Guard individual agents
agent = Agent(role="researcher", goal="...", backstory="...")
agent = guard_agent(agent, api_key="eg_...")

# Or use the standalone guardrail
guardrail = EvalGuardGuardrail(api_key="eg_...", project_id="proj_...")
result = guardrail.check("User input to validate")

# Wrap arbitrary functions
@guardrail.wrap_function
def my_tool(query: str) -> str:
    return do_search(query)

FastAPI Middleware

from evalguardai.fastapi import EvalGuardMiddleware
from fastapi import FastAPI

app = FastAPI()
app.add_middleware(
    EvalGuardMiddleware,
    api_key="eg_...",
    project_id="proj_...",
)

@app.post("/api/chat")
async def chat(request: dict):
    # Automatically guarded -- prompt injection blocked with 403
    return {"response": "..."}

By default, POST requests to paths containing /chat, /completions, /generate, /invoke, or /messages are guarded. Customize with guarded_paths:

app.add_middleware(
    EvalGuardMiddleware,
    api_key="eg_...",
    guarded_paths={"/api/v1/chat", "/api/v1/generate"},
)

For per-route control:

from evalguardai.fastapi import guard_route

@app.post("/api/chat")
@guard_route(api_key="eg_...", project_id="proj_...")
async def chat(request: Request):
    body = await request.json()
    ...

rules= does not scope the check — but on the framework wrappers it is the ON/OFF SWITCH. Two separate facts, and conflating them is what caused a real fail-open:

1. On the wire it is inert. POST /v1/guardrails derives its rule set solely from the project's guardrail_rules rows (or the platform defaults when the project has none) — it has never read a rules field off the request body, and check_input / check_output deliberately strip it before sending (del rules in guardrails.py). Scope a route by pointing it at a project_id whose rules you control. To select attack categories per call, use check_firewall_advanced() (POST /v1/firewall/check), which does honor them.
2. In the framework wrappers it decides whether a check RUNS AT ALL. Every enforcement site in the integrations is gated on if rules: / if input_rules:, so a wrapper built with no rules performs zero guardrail requests and blocks nothing. An earlier version of this note stopped at fact 1 — "passing it changes nothing" — which reads as "you can safely omit it". You cannot: omitting it turns enforcement off.

As of 2.2.2 the wrappers refuse that state instead of accepting it silently: GradioGuard(...) and guarded_chat(...) raise GuardrailNotConfigured when block_on_violation=True (the default) and no rules were supplied, and evalguard.mirascope / evalguard.controlflow already raised the same error for the missing-API-key half. Tracing-only is spelled explicitly — GradioGuard(api_key=..., block_on_violation=False), or the traced_* helpers.

@guard_route scans all four handler idioms — a Pydantic body is read field-by-field, exactly like a raw JSON body:

class ChatRequest(BaseModel):
    prompt: str

@app.post("/api/chat")
@guard_route(api_key="eg_...")
async def chat(body: ChatRequest):      # Pydantic body
    ...

@guard_route(api_key="eg_...")
async def chat(body: dict): ...          # raw JSON body
async def chat(q: str): ...              # text query / body param

Upgrade note (2.2.1). In 2.2.0 and earlier, @guard_route located the request by duck-typing its arguments for an object with both .json and .method. A handler taking a Pydantic body matched nothing and the decorator fell straight through to the handler — no guardrail call, no log line, no warning. If you are on <= 2.2.0 and your handler does not take a Request, that route is not being guarded. 2.2.1 scans those handlers and raises TypeError at import for a signature it cannot guard at all (e.g. async def chat()), so a silently-unguarded route cannot be deployed.

NeMo / Agent Workflows

from evalguardai.nemoclaw import EvalGuardAgent

agent = EvalGuardAgent(api_key="eg_...", agent_name="support-bot")

# Guard any LLM call
result = agent.guarded_call(
    provider="openai",
    messages=[{"role": "user", "content": "Reset my password"}],
    llm_fn=lambda: openai_client.chat.completions.create(
        model="gpt-4", messages=[{"role": "user", "content": "Reset my password"}]
    ),
)

# Multi-step agent sessions
with agent.session("ticket-123") as session:
    session.check("User says: reset my password")
    result = do_llm_call(...)
    session.log_step("password_reset", input="...", output=str(result))

Core Guardrail Client

All framework integrations share the same underlying GuardrailClient:

from evalguardai.guardrails import GuardrailClient

guard = GuardrailClient(
    api_key="eg_...",
    project_id="proj_...",
    timeout=5.0,       # keep low to avoid latency
    fail_open=False,   # fail-closed (default): raise on EvalGuard error so an outage can't silently bypass guardrails
)

# Pre-LLM check. The rule set comes from the project's `guardrail_rules`
# rows — see the `rules=` note above: on THIS client the parameter is inert on
# the wire, but on the framework wrappers it is what turns enforcement on.
result = guard.check_input("user prompt here")
if not result["allowed"]:
    print("Blocked:", result["violations"])

# Post-LLM check
result = guard.check_output("model response here")

# To select attack categories per call, use the firewall route, which
# DOES honor them (this lives on EvalGuard, not GuardrailClient):
from evalguardai import EvalGuard

client = EvalGuard(api_key="eg_...")
fw = client.check_firewall_advanced("user prompt here", rules=["injection", "pii"])

# Fire-and-forget trace
guard.log_trace({"model": "gpt-4", "input": "...", "output": "...", "latency_ms": 120})

Error Handling

All integrations are fail-closed by default: if the EvalGuard API is unreachable, the guardrail check raises (the LLM call is blocked) so an outage cannot silently bypass your guardrails.

fail_open and block_on_violation are different knobs and this section used to conflate them. Reaching for the wrong one leaves you unprotected in exactly the case you were trying to cover:

An absent verdict is not a permissive verdict (2.2.0)

"Cannot render a verdict" is not only an outage. A 200 whose body is {}, {"success": true, "data": null}, {"success": true, "data": {"latencyMs": 3}} or an error envelope {"success": false, …} — an edge-cache error page, a truncated proxy response, a partially-rolled-out server — carries no action and no allowed. Through 2.1.5 the SDK synthesised an allow from it and the unscreened prompt reached the model at every integration.

Those replies now raise evalguard.GuardrailIndeterminate, which takes the same path as an outage: 503 from @guard_route / EvalGuardMiddleware, a refusal everywhere else, and fail_open=True is the only thing that lets one through.

from evalguard import GuardrailIndeterminate, GuardrailViolation

try:
    guard.check_input(user_prompt)
except GuardrailViolation:
    ...   # the firewall answered: BLOCKED
except GuardrailIndeterminate:
    ...   # the firewall answered nothing usable — treat as an outage

Two related bypasses closed in the same release, both of which needed no outage at all: EvalGuardMiddleware used to stream any request body over 2 MiB to your app unguarded (it now refuses with 413; raise max_body_bytes= if your endpoints genuinely take more), and several integrations scanned only the first few thousand characters of a page/result and reported that verdict as the verdict on the whole thing (they now scan it whole, or refuse).

block_on_violation=False does not make an outage pass — the check still raises, because it never produced a verdict to ignore. Only fail_open=True does that:

# Core client — availability over enforcement on an EvalGuard outage
guard = GuardrailClient(api_key="eg_...", fail_open=True)

# FastAPI per-route decorator — same knob, threaded straight through
@app.post("/api/chat")
@guard_route(api_key="eg_...", fail_open=True)
async def chat(request: Request): ...

# Monitor-only: still fails closed on an outage, but a *violation* is
# recorded rather than blocked
client = wrap(OpenAI(), api_key="eg_...", block_on_violation=False)

When a guardrail cannot render a verdict, @guard_route answers 503 with Retry-After and does not invoke your handler. That is deliberately not a 403: an outage is an availability fault, and keeping the two apart means a firewall outage does not read as a spike in blocked attacks on your dashboards.

Catch violations explicitly:

from evalguardai import GuardrailViolation

try:
    response = client.chat.completions.create(...)
except GuardrailViolation as e:
    print(f"Blocked: {e.violations}")

All SDK Methods

Agent Memory Governance

EvalGuard's durable agent memory is a per-session long-term store — write facts with remember_memory, retrieve them by semantic similarity with recall_memory, and clear them with forget_memory:

from evalguardai import EvalGuard

client = EvalGuard(api_key="eg_live_...")

client.remember_memory(
    project_id="proj_...",
    session_key="user-42",
    facts=["Prefers metric units", "Escalate billing questions to a human"],
)
hits = client.recall_memory(
    project_id="proj_...", session_key="user-42", query="what units?"
)["semantic"]
client.forget_memory(project_id="proj_...", session_key="user-42")

Governance puts an org-wide (optionally per-project) policy in front of those durable-memory writes — screening for memory poisoning, requiring human approval on autonomous rewrites, and flagging memories that lack provenance:

# Read the org-wide policy. `org_id` auto-resolves to your default org when
# omitted; `policy` is None until a policy is set.
policy = client.get_agent_memory_governance()["policy"]

# Upsert an org-wide policy. mode -> "off" | "monitor" | "enforce".
client.set_agent_memory_governance(
    mode="enforce",
    enabled=True,
    poison_min_confidence=0.75,        # -> config.thresholds.poisonMinConfidence (0..1)
    require_approval_on_rewrite=True,  # HITL gate on autonomous consolidate/rewrite writes
    require_provenance=True,           # flag any governed memory that lacks a source
)

# Scope a policy to a single project (falls back to the org policy when absent).
client.set_agent_memory_governance(project_id="proj_...", mode="monitor")

# Remove a policy (reverts to no governance).
client.delete_agent_memory_governance()

What the modes do — and what actually enforces. off allows every write; monitor records would-be verdicts but never gates a write; enforce acts (blocks poisoned writes, holds autonomous rewrites for approval). These calls are admin-only server-side and persist the policy row only. Whether enforce actually gates writes is a separate app-layer flag, EVALGUARD_ENFORCE_MEMORY_GOVERNANCE — with it off, a saved enforce policy behaves like monitor (verdicts recorded, no write blocked). org_id is required by the route; the SDK auto-resolves your default org when you omit it. On an org whose governance table has not been migrated yet, the read returns policy: None. Passing a mode other than off/monitor/enforce raises ValueError before any request.

Gateway Guardrail Config

CRUD over the per-project, opt-in gateway_guardrail_config rows the gateway proxy reads to wire guardrail adapters into the hot path. An empty list means the gateway runs its built-in inline firewall only; add rows to layer on the local presets or a partner-vendor adapter (and the reversible PII tokenizer):

# List a project's rows (ordered by priority ascending).
rows = client.list_guardrail_configs(project_id="proj_...")

# Wire a LOCAL preset — makes no external call, so it carries NO secret_ref.
client.upsert_guardrail_config(
    vendor="data-not-instructions",   # a Wave-2 agent guardrail
    on_flag="block",                  # "block" | "redact" | "flag" (default: block)
    check_request=True,
    priority=10,                      # lower runs first
)

# Wire a PARTNER vendor — REQUIRES secret_ref: a UUID for a stored provider-key
# row (the raw vendor key is NEVER sent through this call).
client.upsert_guardrail_config(
    vendor="lakera",
    secret_ref="00000000-0000-0000-0000-000000000000",
    on_flag="redact",
    check_request=True,
    check_response=True,
    tokenize_pii=True,
)

# Delete a row by its id (from list_guardrail_configs), scoped to the project.
client.delete_guardrail_config(config_id=rows[0]["id"], project_id="proj_...")

Local vs. vendor secret rule. The four LOCAL vendors — local-firewall, moderated-firewall, data-not-instructions, tool-call-circuit-breaker (the last two are the Wave-2 agent guardrails) — run in-process and MUST NOT carry a secret_ref; every other (partner-vendor) guardrail REQUIRES one. The SDK enforces both halves up front (raises ValueError before any request), as does the route. Upserts are admin-only, org-scoped, and idempotent on (project_id, vendor) — re-submitting the same vendor updates the row in place. vendor_chain builds a failover chain whose first element must equal vendor (the primary).

Importers

The Python SDK does not ship a dedicated trace-importer method — there is no import_traces / from_promptfoo / from_langsmith call on EvalGuard. Importing historical traces from another platform lives outside this SDK:

• REST: POST /v1/traces/import — body { "platform": "...", "projectId": "<uuid>", "payload": <vendor export JSON> }; returns { inserted, failed, errors, skippedDuplicates }. Requires the editor role, caps each call at 500 spans / 10 MiB (batch larger exports), and dedupes so re-running is safe.
• TypeScript: importTraces(platform, payload) from @evalguard/core.
• CLI: evalguard import:traces --from <platform> <file> — note the colon; the command is registered as import:traces, not import-traces (evalguard import-traces exits 1 with error: unknown command).

Supported platform values — 18 of them: helicone, langfuse, portkey, huggingface, humanloop, vellum, athina, maxim, langsmith, braintrust, deepeval, ragas, giskard, phoenix, mlflow, weave, trulens, opik. That list is SUPPORTED_PLATFORMS in packages/core/src/integrations/trace-importers/index.ts, the single registry behind the REST route, the CLI's --from validation, and importTraces(). For eval-suite migration guides (Promptfoo, DeepEval, Ragas, LangSmith, Braintrust, Humanloop), see the migration docs.

Documentation

Full documentation at evalguard.ai/docs/python-sdk.

License

Apache-2.0 -- see LICENSE for details.