agent-trace-eval

Golden trace regression evaluation for tool-using AI agents.

agent-trace-eval checks agent execution traces against declarative expectations for:

• tool selection (required / forbidden tools)
• tool-call arguments
• tool-call ordering
• multi-agent handoffs
• recovery decisions (retry / fallback / escalate)

It is designed as a small, employer-neutral library you can use in CI to gate agent workflow changes.

Install

pip install agent-trace-eval

For local development:

pip install -e ".[dev]"

Quick start

Run the bundled examples:

agent-trace-eval \
  --cases examples/cases \
  --traces-dir examples/traces \
  --report reports/example-report.md

Trace format

Traces are JSON or YAML documents with an events list. Supported event types include:

• tool_call
• handoff
• recovery_decision
• final_answer

Example:

{
  "case_id": "refund_lookup",
  "events": [
    {
      "type": "tool_call",
      "name": "lookup_order",
      "arguments": { "order_id": "12345" }
    },
    {
      "type": "tool_call",
      "name": "issue_refund",
      "arguments": { "order_id": "12345" }
    }
  ]
}

Case format

Cases are YAML or JSON files with id, description, input, and expect sections:

id: refund_lookup
description: Agent should look up an order before issuing a refund.
expect:
  tools:
    required: [lookup_order, issue_refund]
    forbidden: [delete_account]
  ordering:
    before:
      - first: lookup_order
        second: issue_refund
  arguments:
    issue_refund:
      order_id: "12345"

Python API

from agent_trace_eval import RegressionRunner, render_markdown_report
from agent_trace_eval.loader import load_case, load_trace
from agent_trace_eval.result import SuiteResult

case = load_case("examples/cases/refund_lookup.yaml")
trace = load_trace("examples/traces/refund_lookup.json")

runner = RegressionRunner()
result = runner.run_case(case, trace)
report = render_markdown_report(SuiteResult(case_results=[result]))
print(report)

Related writing

This project complements a series on agent regression testing and release gates:

• Final Answers Are Not Enough (Part 1)
• From Golden Traces to Release Gates (Part 2)

Development

pytest

License

MIT