evalgate-sdk 3.6.0

EvalGate Python SDK — CI for AI behavior. Traces, evaluations, assertions, and regression gates for LLM apps.

evalgate-sdk

evalgate-sdk brings EvalGate's evaluation control plane to Python.

Use it to run assertions, trace agent workflows, inspect platform runs, orchestrate judges, and participate in the same closed-loop evaluation workflow as the TypeScript SDK and the web app.

Install

pip install evalgate-sdk

Optional extras:

pip install "evalgate-sdk[openai]"
pip install "evalgate-sdk[anthropic]"
pip install "evalgate-sdk[langchain]"
pip install "evalgate-sdk[all]"

Quick Start

Local Assertions

from evalgate_sdk import expect

result = expect("The capital of France is Paris.").to_contain("Paris")
print(result.passed)

Platform Client

from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import CreateTraceParams

client = AIEvalClient(api_key="sk-...")

trace = await client.traces.create(
    CreateTraceParams(
        name="support-run",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
    )
)

print(trace.id)

Closed-Loop Workflow

The Python package participates in the same EvalGate loop:

trace -> cluster -> synthesize -> gate -> review -> auto -> ship

Representative CLI commands:

evalgate gate
evalgate cluster --run .evalgate/runs/latest.json
evalgate synthesize --dataset .evalgate/golden/labeled.jsonl
evalgate judge registry
evalgate judge presets

Web App Sync

The Python CLI uses the same sync contracts as the TypeScript SDK. Local labeled JSONL can be pushed into the web app, web-labeled cases can be merged back into the same file, and prompt files can round-trip with stale active-version protection.

# Push local labels into the web app and pull web labels back into JSONL
evalgate sync \
  --project-key support-agent \
  --labeled-dataset .evalgate/golden/labeled.jsonl \
  --pull-labels

# Pull the active web prompt into a local file
evalgate sync \
  --project-key support-agent \
  --prompt-id prompt_support \
  --target-path prompts/support.md \
  --pull-prompt

# Push an edited local prompt as a new web prompt version
evalgate sync \
  --project-key support-agent \
  --prompt-id prompt_support \
  --target-path prompts/support.md \
  --push-prompt

Judge Orchestration

The Python SDK supports the same core judge model as the platform:

• registry-backed judge selection
• saved judge configs
• multi-judge aggregation
• per-run and per-case judge inspection
• structured reasoning and signal breakdowns

CLI

evalgate judge registry
evalgate judge presets

evalgate judge test \
  --provider openai \
  --model gpt-5.2-chat-latest \
  --judge openai:gpt-5.2-chat-latest \
  --judge anthropic:claude-sonnet-4-20250514 \
  --aggregation weighted \
  --prompt-template "Return strict JSON with score, passed, reasoning, and signals." \
  --input "Cancel my subscription" \
  --output "I've canceled your plan effective today."

Client

from evalgate_sdk.types import TestLLMJudgeConfigParams

registry = await client.llm_judge.list_registry()
presets = await client.llm_judge.list_presets()

result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider="openai",
        model="gpt-5.2-chat-latest",
        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
        judges=[
            {
                "id": "primary",
                "type": "llm",
                "provider": "openai",
                "model": "gpt-5.2-chat-latest",
                "weight": 0.6,
            },
            {
                "id": "backup",
                "type": "llm",
                "provider": "anthropic",
                "model": "claude-sonnet-4-20250514",
                "weight": 0.4,
            },
        ],
        aggregation="weighted",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
        behavior="tool_use",
        task_type="support",
    )
)

print(result.result.score, result.result.reasoning)

Enterprise Controls

The Python SDK respects the same platform guardrails:

• provider allowlists
• pre-send PII checks for outbound judge/provider calls
• org-scoped policy
• shared config precedence with the API and web app

Surface Parity

The Python SDK now covers the primary EvalGate workflows used in the product:

• traces and runs
• clustering and synthesis
• gate/check/review loops
• judge registry, presets, configs, results, and comparisons
• workflow and control-plane inspection commands

TypeScript still has a few extra convenience wrappers, but the core control-plane model is shared.

For broader product context, see:

• Root README
• Integration reference
• SDK parity guide