Agent Eval Contract

Pydantic contracts and JSON Schemas for portable agent evaluation records.

Use this package when you are experimenting with agents, harnesses, CI checks, or benchmark runners and need a stable record shape for tasks, runs, scores, failures, and normalized external results. It does not run evaluations, call model providers, store dashboards, or orchestrate agents. It gives those tools a shared contract.

Install

pip install agent-eval-contract

For local development from this repo:

uv sync --dev

Validate A Record

from agent_eval_contract import validate_eval_run

run = validate_eval_run(
    {
        "run_id": "run-login-flow-001",
        "task_id": "task-login-flow-001",
        "harness": "pytest",
        "model": "gpt-5",
        "mode": "autonomous",
        "context_profile": "repo_only",
        "final_status": "success",
        "checks": ["pytest tests/test_auth_redirect.py -q"],
    }
)

print(run.model_dump(mode="json"))

Validation returns typed Pydantic model instances. Invalid records raise pydantic.ValidationError with structured field errors.

CLI

agent-eval-contract validate --kind run --file examples/eval_run.json
agent-eval-contract schemas --output-dir /tmp/agent-eval-contract-schemas
agent-eval-contract fixtures --output-dir /tmp/agent-eval-contract-fixtures
agent-eval-contract normalize --harness terminal-bench --file examples/terminal_bench_result.json --task-id task-login-flow-001 --model gpt-5
agent-eval-contract normalize --harness swe-bench --file examples/swe_bench_result.json

The legacy agent-eval-contract-fixtures command still writes fixture bundles for one release.

What It Provides

• Pydantic models for eval tasks, runs, scores, failures, external results, normalized runs, and fixture manifests
• runtime validators that return typed model instances
• JSON Schema export for all public models
• bundled sample records and markdown templates
• Terminal-Bench and SWE-bench oriented normalization helpers
• a small CLI for validation, schema export, fixture generation, and normalization

Contract Vocabulary

The public core uses generic vocabulary only. Project-specific concepts should live in metadata or a separate adapter package.

• context_profile: repo_only, provided_context, clean_room, tool_augmented, full_workspace
• source: manual, ci, benchmark, production_trace, synthetic
• mode: interactive, autonomous, shadow, replay, benchmark
• final_status: success, partial, failed, abandoned, error

See docs/contract.md, docs/field-reference.md, and docs/adapters.md for the model contract and adapter guidance.

Development

uv run ruff check agent_eval_contract tests
uv run ruff format --check agent_eval_contract tests
uv run basedpyright agent_eval_contract tests
uv run pytest -q
uv build --out-dir /tmp/agent-eval-contract-dist