Skip to main content

understudy: Scenario Testing for AI Agents

PyPI version PyPI Downloads Python 3.12+ Documentation License: MIT

Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, and handoffs, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.

How It Works

Testing with understudy is 4 steps:

  1. Wrap your agent — Adapt your agent (ADK, LangGraph, LangChain, HTTP) to understudy's interface
  2. Mock your tools — Register handlers that return test data instead of calling real services
  3. Write scenes — YAML files defining what the simulated user wants and what you expect
  4. Run and assert — Execute simulations, check traces, generate reports

The key insight: assert against the trace, not the prose. Don't check what the agent said—check what it did (tool calls).

Two Evaluation Paradigms

Conversational Agent Evaluation

Simulate multi-turn conversations with personas to test dialog agents.

  • Use case: Customer service bots, assistants, chatbots
  • Assert on tool calls: trace.called("tool_name")

Agentic Flow Evaluation

Evaluate autonomous agents executing multi-step tasks.

  • Use case: Code agents, research agents, task automation
  • Assert on actions: trace.performed("action")

See examples/README.md for complete examples of both paradigms.

See real examples:

State injection and next-turn tests

Restore a completed checkpoint, inject synthetic history/state, or replay through exchange X, then execute and assert only against exchange X+1. ADK and LangGraph support isolated checkpoint forks; LangChainApp builds an instrumented LangChain agent with tool middleware. Traces retain nested invocation identities.

See the state-testing guide for run_turn, replay_to, scoped tool mocks, pytest fixtures, CLI commands, and framework-specific restoration limits.

Installation

pip install "understudy[all]"

Quick Start

1. Wrap your agent

from understudy.adk import ADKApp
from my_agent import agent

app = ADKApp(agent=agent)

2. Mock your tools

Your agent has tools that call external services. Mock them for testing:

from understudy.mocks import MockToolkit

mocks = MockToolkit()


@mocks.handle("lookup_order")
def lookup_order(order_id: str) -> dict:
    return {
        "order_id": order_id,
        "items": [{"sku": "BACKPACK-01", "quantity": 1}],
        "status": "delivered",
    }


@mocks.handle("create_return")
def create_return(order_id: str, item_sku: str, reason: str) -> dict:
    return {"return_id": "RET-001", "status": "created"}

3. Write a scene

Create scenes/return_backpack.yaml:

id: return_eligible_backpack
description: Customer wants to return a backpack

starting_prompt: "I'd like to return an item please."
conversation_plan: |
  Goal: Return the hiking backpack from order ORD-10031.
  - Provide order ID when asked
  - Return reason: too small

persona: cooperative
max_turns: 15

expectations:
  required_tools:
    - lookup_order
    - create_return
  forbidden_tools:
    - issue_refund

4. Run simulation

from understudy import Scene, run

scene = Scene.from_file("scenes/return_backpack.yaml")
trace = run(app, scene, mocks=mocks)

assert trace.called("lookup_order")
assert trace.called("create_return")
assert not trace.called("issue_refund")

Or with pytest (define app and mocks fixtures in conftest.py):

pytest test_returns.py -v

Suites and Batch Runs

Run multiple scenes with multiple simulations per scene:

from understudy import Suite, RunStorage

suite = Suite.from_directory("scenes/")
storage = RunStorage()

# Run each scene 3 times and tag for comparison
results = suite.run(
    app,
    mocks=mocks,
    storage=storage,
    n_sims=3,
    tags={"version": "v1"},
)
print(f"{results.pass_count}/{len(results.results)} passed")

Simulation and Evaluation

Understudy separates simulation (generating traces) from evaluation (checking traces). Use together or separately:

Combined (most common)

understudy run \
  --app mymodule:agent_app \
  --scene ./scenes/ \
  --n-sims 3 \
  --junit results.xml

Separate workflows

Generate traces only:

understudy simulate \
  --app mymodule:agent_app \
  --scenes ./scenes/ \
  --output ./traces/ \
  --n-sims 3

Evaluate existing traces:

understudy evaluate \
  --traces ./traces/ \
  --output ./results/ \
  --junit results.xml

Python API:

from understudy import simulate_batch, evaluate_batch

# Generate traces
traces = simulate_batch(
    app=agent_app,
    scenes="./scenes/",
    n_sims=3,
    output="./traces/",
)

# Evaluate later
results = evaluate_batch(
    traces="./traces/",
    output="./results/",
)

CLI Commands

# Run simulations
understudy run --app mymodule:app --scene ./scenes/
understudy simulate --app mymodule:app --scenes ./scenes/
understudy evaluate --traces ./traces/

# View results
understudy list
understudy show <run_id>
understudy summary

# Compare runs by tag
understudy compare --tag version --before v1 --after v2

# Generate reports
understudy report -o report.html
understudy compare --tag version --before v1 --after v2 --html comparison.html

# Interactive browser
understudy serve --port 8080

# HTTP simulator server (for browser/UI testing)
understudy serve-api --port 8000

# Cleanup
understudy delete <run_id>
understudy clear

LLM Judges

For qualities that can't be checked deterministically:

from understudy.judges import Judge

empathy_judge = Judge(
    rubric="The agent acknowledged frustration and was empathetic while enforcing policy.",
    samples=5,
)

result = empathy_judge.evaluate(trace)
assert result.score == 1

Built-in rubrics:

from understudy.prompts import (
    TOOL_USAGE_CORRECTNESS,
    POLICY_COMPLIANCE,
    TONE_EMPATHY,
    ADVERSARIAL_ROBUSTNESS,
    TASK_COMPLETION,
)

Report Contents

The understudy summary command shows:

  • Pass rate — percentage of scenes that passed all expectations
  • Avg turns — average conversation length
  • Tool usage — distribution of tool calls across runs
  • Agents — which agents were invoked

The HTML report (understudy report) includes:

  • All metrics above
  • Full conversation transcripts
  • Tool call details with arguments
  • Expectation check results
  • Judge evaluation results (when used)

Documentation

See the full documentation for:

License

MIT

Metadata

Release files for understudy 0.6.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for understudy 0.6.0
File Size Uploaded Provenance Actions
understudy-0.6.0.tar.gz 93.3 kB With provenance Details for understudy-0.6.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for understudy 0.6.0
File Interpreter ABI Platform Actions
understudy-0.6.0-py3-none-any.whl Python 3 none any Details for understudy-0.6.0-py3-none-any.whl

Total release size: 205.8 kB

Release files / understudy-0.6.0.tar.gz

Download URL understudy-0.6.0.tar.gz
Size 93.3 kB
Tags Source
SHA-256 checksum
How to use checksums
6f7737986b4114b60b89f25effe2393679c1dc8b6d94e35cec6613b8c00ddc87
BLAKE2b-256 checksum
How to use checksums
ca77b12b02c76c356c0b3467d4409894e493139f2f1281dde0141fbd2391ca68
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log

Release files / understudy-0.6.0-py3-none-any.whl

Download URL understudy-0.6.0-py3-none-any.whl
Size 112.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8e4d9fce84eba0f7cd1e280c2d790cac05d1384fc242193997a2de51acd8542d
BLAKE2b-256 checksum
How to use checksums
0780524877812070aca673faec1df990b42a20bbfec0d33fa5c8097f2fa1f393
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log

Security for understudy 0.6.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for understudy-0.6.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page