evalgate-sdk 3.5.0

EvalGate Python SDK — CI for AI behavior. Traces, evaluations, assertions, and regression gates for LLM apps.

evalgate-sdk

evalgate-sdk brings EvalGate's evaluation control plane to Python.

Use it to run assertions, trace agent workflows, inspect platform runs, orchestrate judges, and participate in the same closed-loop evaluation workflow as the TypeScript SDK and the web app.

Install

pip install evalgate-sdk
pip install "evalgate-sdk[cli]"

Optional extras:

pip install "evalgate-sdk[openai]"
pip install "evalgate-sdk[anthropic]"
pip install "evalgate-sdk[langchain]"
pip install "evalgate-sdk[all]"

Quick Start

Local Assertions

from evalgate_sdk import expect

result = expect("The capital of France is Paris.").to_contain("Paris")
print(result.passed)

Platform Client

from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import CreateTraceParams

client = AIEvalClient(api_key="sk-...")

trace = await client.traces.create(
    CreateTraceParams(
        name="support-run",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
    )
)

print(trace.id)

Closed-Loop Workflow

The Python package participates in the same EvalGate loop:

trace -> cluster -> synthesize -> gate -> review -> auto -> ship

Representative CLI commands:

evalgate gate
evalgate cluster --run .evalgate/runs/latest.json
evalgate synthesize --dataset .evalgate/golden/labeled.jsonl
evalgate judge registry
evalgate judge presets

Judge Orchestration

The Python SDK supports the same core judge model as the platform:

• registry-backed judge selection
• saved judge configs
• multi-judge aggregation
• per-run and per-case judge inspection
• structured reasoning and signal breakdowns

CLI

evalgate judge registry
evalgate judge presets

evalgate judge test \
  --provider openai \
  --model gpt-5.2-chat-latest \
  --judge openai:gpt-5.2-chat-latest \
  --judge anthropic:claude-sonnet-4-20250514 \
  --aggregation weighted \
  --prompt-template "Return strict JSON with score, passed, reasoning, and signals." \
  --input "Cancel my subscription" \
  --output "I've canceled your plan effective today."

Client

from evalgate_sdk.types import TestLLMJudgeConfigParams

registry = await client.llm_judge.list_registry()
presets = await client.llm_judge.list_presets()

result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider="openai",
        model="gpt-5.2-chat-latest",
        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
        judges=[
            {
                "id": "primary",
                "type": "llm",
                "provider": "openai",
                "model": "gpt-5.2-chat-latest",
                "weight": 0.6,
            },
            {
                "id": "backup",
                "type": "llm",
                "provider": "anthropic",
                "model": "claude-sonnet-4-20250514",
                "weight": 0.4,
            },
        ],
        aggregation="weighted",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
        behavior="tool_use",
        task_type="support",
    )
)

print(result.result.score, result.result.reasoning)

Enterprise Controls

The Python SDK respects the same platform guardrails:

• provider allowlists
• pre-send PII checks for outbound judge/provider calls
• org-scoped policy
• shared config precedence with the API and web app

Surface Parity

The Python SDK now covers the primary EvalGate workflows used in the product:

• traces and runs
• clustering and synthesis
• gate/check/review loops
• judge registry, presets, configs, results, and comparisons
• workflow and control-plane inspection commands

TypeScript still has a few extra convenience wrappers, but the core control-plane model is shared.

For broader product context, see:

• Root README
• Integration reference
• SDK parity guide