evalgate-sdk

evalgate-sdk helps Python teams decide whether an AI change is safe to ship. It combines local assertions and release gating, Python client/library surfaces, CLI workflows, optional hosted collaboration and evidence, and CI integration.

Install

pip install evalgate-sdk

Optional extras:

pip install "evalgate-sdk[openai]"
pip install "evalgate-sdk[anthropic]"
pip install "evalgate-sdk[langchain]"
pip install "evalgate-sdk[all]"

all installs the OpenAI, Anthropic, LangChain, and CLI dependencies. The CrewAI and AutoGen adapters do not install those frameworks; install the framework distribution used by your application separately.

Navigate by product outcome instead of memorizing the flat command catalog:

evalgate capabilities
evalgate capabilities playground
evalgate capabilities --format json

The exported PRODUCT_CAPABILITY_CONTRACT is generated from the same versioned workflow map used by the Python and TypeScript CLIs.

Availability

The lifecycle below describes the checked-in source contract. PyPI publication is separately governed and the registry can lag the source tree; run evalgate --version and check the public feature-status page before relying on a source-only command. Use evalgate capabilities --format json for the installed package's machine-readable workflow and exit-code contract.

Start in a cloud-agent sandbox — no account or browser required

evalgate init --local --format json
evalgate init --local --apply --format json
evalgate baseline update
evalgate gate

Here, local names the execution/evidence boundary for the current checkout, not a requirement that a person be on a laptop. It requires no account, API key, organization, GitHub remote, browser, interactive TTY, or network call. It does not invent hosted identity. Review and commit durable configuration/spec files before accepting the baseline; generated reports and run artifacts follow managed ignore rules. --preview never writes or prompts, and JSON/non-interactive execution remains preview-only without explicit --apply.

The daily loop is change -> evalgate gate -> PASS / WARN / BLOCK + evidence. Use evalgate status --json to read local readiness independently from hosted identity and cloud evidence. Status exit zero means the document was produced, not that every readiness dimension passed.

Add hosted collaboration

evalgate login
evalgate link
evalgate status

Current source stores the revocable human CLI session outside the repository in the operating-system user configuration directory: %APPDATA%\evalgate\config.json on Windows, ~/Library/Application Support/evalgate/config.json on macOS, and ${XDG_CONFIG_HOME:-~/.config}/evalgate/config.json on Linux. An absolute EVALGATE_CONFIG_HOME can override that directory for managed environments. link establishes the durable organization, GitHub repository, and selected repository-relative root, then pins the cloud activation snapshot to the exact local commit. It writes only non-secret version 2 context to .evalgate/project.json. Branch and commit movement do not unlink the project. status reports durable link, checkout relation, cloud-snapshot currency, and local/cloud readiness separately. A checkout two commits behind main remains linked and locally gate-ready; cloud intake may target that exact SHA when GitHub proves it remains reachable. EvalGate never silently substitutes the remote head. Repository files cannot provide an API key or redirect a saved/environment credential to another API origin. Alternate origins require HTTPS except localhost and loopback development. CI should inject a separately managed EVALGATE_API_KEY through its secret store; linking never creates or replaces that automation credential. An API key authorizes automation against an already established hosted context; it is not a bootstrap shortcut for human login, repository linking, or local initialization.

Version 3.7.9 source briefly used .evalgate/config.json. Current source ignores that legacy file: run evalgate login, delete it, and rotate the key if it may have entered Git history.

Hosted local gate setup

evalgate init
evalgate baseline update

Interactive init previews the scaffold and asks before writing. JSON and non-interactive invocations remain preview-only; explicit init --preview never prompts or writes, and automation must pass evalgate init --apply explicitly. A valid safe version 2 project context is required for hosted initialization, but commit drift does not invalidate it and linked projects may initialize offline. Then use evalgate gate before shipping a change. Linking never mints an automation credential. Existing local specs, configuration, and baselines remain in place when a local project is later linked.

Migrating from 3.8.x source

Existing local project state remains usable; do not reinitialize or replace a valid baseline just to adopt the new lifecycle.

3.8.x	Current path
auth status	whoami
auth configure --api-key ...	Human: login; automation: environment credential with an existing hosted context; local: init --local
setup	login, then link
setup status	status
start	No direct alias: use init/init --local for setup, gate for a decision, or watch for continuous reruns
verify	status, then gate

Removed commands provide targeted guidance rather than reviving repository secret storage.

Optional product understanding (Experimental current source)

understand is optional and is not required for initialization, baseline acceptance, or a release decision:

evalgate understand --format json
evalgate understand --apply

It reads bounded local evidence and produces provisional product context without network/model calls, application execution, baseline mutation, or promotion.

On-demand application health

The same local path works for ordinary applications such as APIs, web apps, webhook consumers, background jobs, and CLI tools. Create a small evalgate.application.json manifest with safe HTTP or command checks and run:

evalgate health --config evalgate.application.json --format github

The report measures bounded latency and errors sequentially, redacts failure text, and does not keep a collector, daemon, or cloud connection alive. The reviewed evalgate init scaffold installs the evalgate-application-health repository Skill so a coding agent can infer checks and prepare a reviewed PR rationale only when called.

Local Assertions

from evalgate_sdk import expect

result = expect("The capital of France is Paris.").to_contain("Paris")
print(result.passed)

Platform Client

import asyncio

from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import CreateTraceParams

async def main() -> None:
    async with AIEvalClient(api_key="sk-...") as client:
        trace = await client.traces.create(
            CreateTraceParams(
                name="support-run",
                input="Cancel my subscription",
                output="I've canceled your plan effective today.",
            )
        )

        print(trace.id)

asyncio.run(main())

Closed-Loop Workflow

The Python package participates in the same EvalGate loop:

trace -> cluster -> synthesize -> gate -> review -> auto -> ship

Representative CLI commands:

evalgate gate
evalgate cluster --run .evalgate/runs/latest.json
evalgate synthesize --dataset .evalgate/golden/labeled.jsonl
evalgate auto run --objective tone_mismatch --budget 3
evalgate auto daemon --objective tone_mismatch --cycles 5
evalgate judge registry
evalgate judge presets

Connect a coding client

Install EvalGate's repository Skill using the client-native layout where one is known. Preview is the default; --apply is the only write authority:

evalgate connect --client codex
evalgate connect --client codex --apply

Use claude-code, cursor, cowork, or generic for other clients. Setup writes local client files after the repository's signed-in EvalGate install. The generated .evalgate/agent-clients.json records the metrics-only export path and next capture/sync commands; EvalGate does not read provider transcripts or start a collector.

Web App Sync

The Python CLI uses the same sync contracts as the TypeScript SDK. Local labeled JSONL can be pushed into the web app, web-labeled cases can be merged back into the same file, and prompt files can round-trip with stale active-version protection.

# Push local labels into the web app and pull web labels back into JSONL
evalgate sync \
  --project-key support-agent \
  --labeled-dataset .evalgate/golden/labeled.jsonl \
  --pull-labels

# Pull the active web prompt into a local file
evalgate sync \
  --project-key support-agent \
  --prompt-id prompt_support \
  --target-path prompts/support.md \
  --pull-prompt

# Push an edited local prompt as a new web prompt version
evalgate sync \
  --project-key support-agent \
  --prompt-id prompt_support \
  --target-path prompts/support.md \
  --push-prompt

Product Workflows

The purpose-built client and CLI cover the same governed workflows as the web app, including pack installation, exact-commit repository intelligence, Playground case authoring, Prompt Hub, and Dataset Hub.

evalgate repo repositories
evalgate repo scan --repository 42 --head-sha <sha>
evalgate repo ask --repository 42 --question "What AI exists?"
evalgate packs list --domain coding
evalgate packs install coding-agent-release-safety
evalgate playground from-trace --evaluation 7 --trace 123
evalgate prompts --help
evalgate datasets --help

repositories = await client.repository_intelligence.list_repositories()
scan = await client.repository_intelligence.start_scan(
    42,
    idempotency_key="release-42",
    head_sha="a" * 40,
)
answer = await client.repository_intelligence.answer_question(
    42,
    question_text="What models, agents, tools, and evals exist?",
    graph_version_id=scan["scanRun"]["graphVersionId"],
)
packs = await client.eval_packs.list(domain="coding")

Repository scanning reads a bounded immutable GitHub tree and does not execute repository code. Source detections and pack recommendations include evidence locators and completeness state; they are not represented as runtime proof.

Judge Orchestration

The Python SDK supports the same core judge model as the platform:

• registry-backed judge selection
• saved judge configs
• multi-judge aggregation
• per-run and per-case judge inspection
• structured reasoning and signal breakdowns

CLI

evalgate judge registry
evalgate judge presets

evalgate judge test \
  --provider openai \
  --model gpt-5.2-chat-latest \
  --judge openai:gpt-5.2-chat-latest \
  --judge anthropic:claude-sonnet-4-20250514 \
  --aggregation weighted \
  --prompt-template "Return strict JSON with score, passed, reasoning, and signals." \
  --input "Cancel my subscription" \
  --output "I've canceled your plan effective today."

Client

from evalgate_sdk.types import TestLLMJudgeConfigParams

registry = await client.llm_judge.list_registry()
presets = await client.llm_judge.list_presets()

result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider="openai",
        model="gpt-5.2-chat-latest",
        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
        judges=[
            {
                "id": "primary",
                "type": "llm",
                "provider": "openai",
                "model": "gpt-5.2-chat-latest",
                "weight": 0.6,
            },
            {
                "id": "backup",
                "type": "llm",
                "provider": "anthropic",
                "model": "claude-sonnet-4-20250514",
                "weight": 0.4,
            },
        ],
        aggregation="weighted",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
        behavior="tool_use",
        task_type="support",
    )
)

print(result.result.score, result.result.reasoning)

Enterprise Controls

The Python SDK respects the same platform guardrails:

• provider allowlists
• pre-send PII checks for outbound judge/provider calls
• org-scoped policy
• shared config precedence with the API and web app

Cross-Language Contract

The generated OpenApiClient is the cross-language operation surface. It validates stable operation IDs, required path/query/header parameters, required bodies, idempotency keys, cancellation, and bounded pagination from the same OpenAPI registry as TypeScript and the CLI.

from evalgate_sdk.openapi_client import OpenApiClient

async with OpenApiClient(api_key="sk-...") as api:
    versions = await api.request(
        "get_evaluations_id_prompts_promptId_versions",
        path={"id": 42, "promptId": "support/primary"},
    )

Purpose-built Python and TypeScript clients are intentionally documented separately and are not assumed to be one-for-one. See the generated parity report for the structural operation inventory.

For broader product context, see:

• Root README
• Python SDK guide
• Generated SDK parity report
• Feature availability