understudy 0.1.0

Simulation and trace-based evaluation for agentic systems

understudy

Test your AI agents with simulated users.

Installation

pip install understudy[all]

Quick Start

1. Wrap your agent

from understudy.adk import ADKApp
from my_agent import agent

app = ADKApp(agent=agent)

2. Mock your tools

Your agent has tools that call external services. Mock them for testing:

from understudy.mocks import MockToolkit

mocks = MockToolkit()

@mocks.handle("lookup_order")
def lookup_order(order_id: str) -> dict:
    return {"order_id": order_id, "items": [...], "status": "delivered"}

@mocks.handle("create_return")
def create_return(order_id: str, item_sku: str, reason: str) -> dict:
    return {"return_id": "RET-001", "status": "created"}

3. Write a scene

Create scenes/return_backpack.yaml:

id: return_eligible_backpack
description: Customer wants to return a backpack

starting_prompt: "I'd like to return an item please."
conversation_plan: |
  Goal: Return the hiking backpack from order ORD-10031.
  - Provide order ID when asked
  - Return reason: too small

persona: cooperative
max_turns: 15

expectations:
  required_tools:
    - lookup_order
    - create_return
  allowed_terminal_states:
    - return_created

4. Run simulation

from understudy import Scene, run, check

scene = Scene.from_file("scenes/return_backpack.yaml")
trace = run(app, scene, mocks=mocks)

assert trace.called("lookup_order")
assert trace.called("create_return")
assert trace.terminal_state == "return_created"

Or with pytest (define app and mocks fixtures in conftest.py):

pytest test_returns.py -v

CLI Commands

After running simulations, use the CLI to inspect results:

# List all saved runs
understudy list

# Show aggregate metrics (pass rate, avg turns, tool usage, terminal states)
understudy summary

# Show details for a specific run
understudy show <run_id>

# Generate static HTML report
understudy report --output report.html

# Start interactive report browser
understudy serve --port 8080

# Delete runs
understudy delete <run_id>
understudy clear

LLM Judges

For qualities that can't be checked deterministically:

from understudy.judges import Judge

empathy_judge = Judge(
    rubric="The agent acknowledged frustration and was empathetic while enforcing policy.",
    samples=5,
)

result = empathy_judge.evaluate(trace)
assert result.score == 1

Built-in rubrics:

from understudy.judges import (
    TOOL_USAGE_CORRECTNESS,
    POLICY_COMPLIANCE,
    TONE_EMPATHY,
    ADVERSARIAL_ROBUSTNESS,
    TASK_COMPLETION,
)

Report Contents

The understudy summary command shows:

• Pass rate - percentage of scenes that passed all expectations
• Avg turns - average conversation length
• Tool usage - distribution of tool calls across runs
• Terminal states - breakdown of how conversations ended
• Agents - which agents were invoked

The HTML report (understudy report) includes:

• All metrics above
• Full conversation transcripts
• Tool call details with arguments
• Expectation check results
• Judge evaluation results (when used)

Documentation

See the full documentation for:

• Installation guide
• Writing scenes
• ADK integration
• HTTP client for deployed agents
• API reference

License

MIT