Skip to main content

pytest-agent-behavior

Deterministic pytest assertions for AI agent tool calls, approval ordering, and repeated-call limits.

pytest-agent-behavior evaluates a trace that your agent code already produced. It makes no LLM calls and needs no API key, service, dashboard, or framework plugin. Write ordinary pytest tests that catch behavioral regressions in tool selection, approval order, call counts, and repeated identical calls.

Version: 0.1.0

Install

From PyPI:

python -m pip install pytest-agent-behavior

From a local checkout, for development:

python -m pip install .

For development, including tests, linters, and build tools:

python -m pip install -e ".[dev]"

The package registers the pytest plugin entry point agent_behavior automatically. No -p flag or pytest_plugins entry is needed.

Quickstart

from pytest_agent_behavior import Trace


def test_email_requires_approval(agent_assertions):
    trace = Trace.from_dict(
        {
            "schema_version": 1,
            "events": [
                {
                    "type": "approval",
                    "call_id": "c1",
                    "tool": "send_email",
                    "decision": "approved",
                },
                {
                    "type": "tool_call",
                    "call_id": "c1",
                    "tool": "send_email",
                    "arguments": {"to": "fake@example.com", "body": "Hello"},
                },
            ],
        }
    )

    check = agent_assertions(trace)
    check.called("send_email", times=1)
    check.requires_approval("send_email")

Run it with pytest. This quickstart is a passing example. The agent_assertions fixture returns a factory that accepts a validated Trace. It deliberately does not accept dictionaries or paths: call Trace.from_dict, Trace.from_json, or Trace.from_path explicitly so conversion is visible in the test.

All assertion methods return None on success and raise AgentAssertionError (a subclass of AssertionError) on a behavioral failure. Invalid parameter types raise TypeError; valid types with invalid values, such as a negative count or empty tool name, raise ValueError.

Trace format

{
  "schema_version": 1,
  "events": [
    {
      "type": "approval",
      "call_id": "c1",
      "tool": "send_email",
      "decision": "approved"
    },
    {
      "type": "tool_call",
      "call_id": "c1",
      "tool": "send_email",
      "arguments": {
        "to": "example@example.com",
        "body": "Hello"
      }
    }
  ]
}
  • schema_version must be the integer 1. Python True is rejected even though it compares equal to 1.
  • events is an ordered JSON array. Position is the only ordering signal; timestamps are not inferred or supported. Public event indices are zero-based.
  • A tool_call has exact type "tool_call", nonempty string call_id and tool, and a required JSON object arguments.
  • An approval has exact type "approval", nonempty string call_id and tool, and decision exactly "approved" or "denied".
  • Every tool_call.call_id must be unique in a trace. Multiple approvals for one call_id are allowed. Approval-only traces and approvals without a subsequent call are valid.
  • Unknown fields, unknown event types, unsupported versions, non-JSON values, non-finite numbers, non-string object keys, and cyclic values raise TraceValidationError with a path such as events[2].arguments: expected a JSON object.
  • Empty traces and empty arguments objects are valid.
  • A tool call records an observed invocation attempt. It does not imply the external operation succeeded.

The published JSON Schema is at src/pytest_agent_behavior/schema/trace.schema.json. The schema cannot express unique tool_call.call_id values or Python's exact type distinctions; those are enforced by the Python loader. See docs/trace-format.md for the full format and cross-event constraints.

Assertions

called(tool, *, times=None)

Matches tool names exactly and case-sensitively and counts only tool_call events. times=None requires at least one matching call. times=N requires exactly N; times=0 is valid.

not_called(tool)

Passes only when no matching tool_call event exists. Behaviorally equivalent to called(tool, times=0). Failure messages include matching event indices.

requires_approval(tool)

For every tool_call matching tool, finds approvals before that call with the same call_id and the same tool. The last such preceding approval must be "approved".

  • No matching preceding approval: failure.
  • Approval after the call: does not authorize the earlier call.
  • Approval for another call_id or another tool: does not authorize.
  • approved then denied before the call: failure.
  • denied then approved before the call: success.
  • One call's approval never authorizes another call.
  • If the selected tool never appears, the assertion passes vacuously. Combine it with called(tool) when existence is also required.

Approvals are observed authorization metadata. The package cannot prove who supplied an approval or that an approval actually blocked execution.

max_calls(limit, *, tool=None)

Counts all tool_call events when tool is None, otherwise only exact matches. Passes when the count is at most limit. Approval events never count. limit=0 is valid.

max_repeated_calls(limit, *, tool=None)

Counts total occurrences of each distinct (tool, arguments) signature across the entire trace, not just consecutive repetitions. The assertion fails if any signature count exceeds limit.

  • Object key order is ignored, at every nesting level.
  • Array order matters.
  • Different tool names create different signatures.
  • Different call_id values do not distinguish otherwise identical calls.
  • JSON types are strict: true, 1, and 1.0 are different argument values.
  • Failure messages identify tools, counts, and event indices but never print argument values.

This is a repeated-call threshold check. It does not prove an agent is stuck in a loop or that a retry was inappropriate.

See docs/api.md for constructor details, exception contracts, and privacy notes.

Documentation

A passing test

examples/demo_agents.py contains two deterministic demo agents. They call ordinary local Python functions through a small manual recorder; there are no network requests, LLM calls, or real side effects. The passing agent produces its trace through execution:

from demo_agents import run_passing_agent


def test_passing_demo_agent(agent_assertions):
    trace = run_passing_agent()
    check = agent_assertions(trace)

    check.called("lookup_order", times=1)
    check.called("send_email", times=1)
    check.not_called("issue_refund")
    check.requires_approval("send_email")
    check.max_calls(2)
    check.max_repeated_calls(1)

Run the passing demo:

pytest examples/tests/test_passing_agent.py -q

A deliberately failing test

The faulty demo agent sends four identical emails without approvals. Its intentionally failing checks live in examples/failing_demo.py, which is not named test_*.py and therefore stays out of normal collection.

pytest examples/failing_demo.py -q --tb=short

This command exits with status 1. Actual output from this tree (CPython 3.11.0, pytest 9.2.0):

FF                                                                       [100%]
=================================== FAILURES ===================================
________________ test_faulty_agent_exceeds_repeated_call_limit _________________
examples/failing_demo.py:18: in test_faulty_agent_exceeds_repeated_call_limit
    check.max_repeated_calls(2)
...
E   pytest_agent_behavior.exceptions.AgentAssertionError: Agent behavior assertion failed: max_repeated_calls
E   tool: (any)
E   expected: at most 2 call(s) per distinct (tool, arguments) signature
E   observed: 1 signature(s) exceed the limit (showing 1)
E     tool: send_email
E     count: 4
E     event indices: [0, 1, 2, 3]
______________________ test_faulty_agent_has_no_approval _______________________
...
E   pytest_agent_behavior.exceptions.AgentAssertionError: Agent behavior assertion failed: requires_approval
E   tool: send_email
E   unauthorized calls: 4 (showing 4)
E     event index: 0
E     call_id: c1
E     expected: a preceding matching approval for the same call_id and tool with decision approved
E     observed: no matching preceding approval
...
2 failed in 0.01s

The package's own failure text does not include argument values. Pytest itself may display source lines or locals, which can contain test literals; see Privacy.

Manual instrumentation

Record the attempt immediately before you invoke a tool. Use fake arguments in tests and examples. Approval events must be recorded before the call they are meant to authorize.

from pytest_agent_behavior import Approval, ToolCall, Trace


def run_agent(send_email):
    events = []

    call_id = "c1"
    events.append(Approval(call_id, "send_email", "approved"))
    events.append(
        ToolCall(
            call_id,
            "send_email",
            {"to": "fake@example.com", "body": "Synthetic message"},
        )
    )
    send_email("fake@example.com", "Synthetic message")  # the observed attempt

    events.append(ToolCall("c2", "lookup_order", {"order_id": "order-42"}))
    return Trace(events)

The tool call is recorded regardless of whether the external call later succeeds or raises. This library does not wrap, intercept, or prevent that call.

What this package does and does not do

It does:

  • validate and load supplied sequential traces;
  • evaluate deterministic assertions in ordinary pytest tests;
  • report bounded, index-based diagnostics without argument values.

It does not:

  • make live LLM calls, require API keys, send telemetry, or contact a service;
  • instrument or monkey-patch an agent framework;
  • intercept or prevent tool execution;
  • prove an agent is safe, correct, or compliant;
  • assess answer quality or perform semantic similarity/LLM-as-judge scoring;
  • provide exactly-once execution guarantees;
  • model parallel or multi-agent execution.

Traces are single-run and sequential. Event order is exactly array order. The format has no timestamps, concurrency, causal graph, or cross-run semantics. Interleaved calls from parallel workers cannot be represented faithfully in this release.

Privacy

  • Library-generated assertion failures identify assertion, tool, counts, event indices, and call_id where useful. They do not include argument values or full events.
  • ToolCall.__repr__ omits argument values; Trace.__repr__ is a short summary rather than a dump of events.
  • Validation errors identify paths and value types, not the offending values.
  • to_dict() and to_json() intentionally contain arguments because they serialize the trace. Treat trace files and test literals as potentially sensitive.
  • Pytest can display user source code, assertion expressions, or locals depending on traceback options and plugins. The library cannot promise complete redaction of pytest's output.

There is no telemetry, networking, or background work.

Comparison with alternatives

  • Handwritten assertions are often enough for one simple count. They give total control but tend to duplicate trace validation, approval-order logic, strict JSON signature handling, and diagnostic formatting. This package is a small local helper for those recurring checks.
  • Broader experiment/evaluation platforms commonly manage datasets, hosted tracing, scoring, and LLM-judge workflows. This package is not an evaluation platform, has no hosted component, and performs no semantic scoring.
  • Runtime policy or approval gateways can block execution. This package only evaluates an already-observed trace and cannot provide that guarantee.

Compatibility and development

The package requires Python 3.10+ and declares pytest>=8.3,<10 as its only runtime dependency.

Locally tested for 0.1.0, on macOS arm64 against the checked-out source:

  • CPython 3.10.18, 3.11.0, 3.12.11, 3.13.6, and 3.14.0rc1 with pytest 9.2.0: 178 passed;
  • CPython 3.11.0 with pytest 8.3.5: 178 passed;
  • CPython 3.15.0 with pytest 9.2.0: 178 passed.

GitHub Actions CI passed on Ubuntu for CPython 3.10, 3.11, 3.12, 3.13, and 3.14 with pytest 9.2.0, plus CPython 3.11 with pytest 8.3.5. Each matrix job reported 178 passed. See the repository's Actions runs for the exact release commit. The local 3.14 result used a release candidate; the CI 3.14 result used the stable setup-python build.

Local verification commands:

python -m pip install -e ".[dev]"
pytest -q
ruff check .
ruff format --check .
mypy
python -m build
python -m twine check dist/*

The release process, including TestPyPI/PyPI and Trusted Publishing guidance, is in docs/release-checklist.md.

Maintained by Varun Talari.

License

MIT. See LICENSE.

Metadata

Release files for pytest-agent-behavior 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pytest-agent-behavior 0.1.0
File Size Uploaded Provenance Actions
pytest_agent_behavior-0.1.0.tar.gz 33.7 kB With provenance Details for pytest_agent_behavior-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pytest-agent-behavior 0.1.0
File Interpreter ABI Platform Actions
pytest_agent_behavior-0.1.0-py3-none-any.whl Python 3 none any Details for pytest_agent_behavior-0.1.0-py3-none-any.whl

Total release size: 50.9 kB

Release files / pytest_agent_behavior-0.1.0.tar.gz

Download URL pytest_agent_behavior-0.1.0.tar.gz
Size 33.7 kB
Tags Source
SHA-256 checksum
How to use checksums
b23c2597125a77385562a05ed9a76979906e1165dc9f49de9dce74324a4a00d2
BLAKE2b-256 checksum
How to use checksums
c352a42e6dc49a2172dba4de33a2ca68052aa798ef0e53bbccf2c8851e9fa684
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

Release files / pytest_agent_behavior-0.1.0-py3-none-any.whl

Download URL pytest_agent_behavior-0.1.0-py3-none-any.whl
Size 17.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
959be2a9f04f7224605f8c4d064a98e4e8b580b1ce78dcc53b2dc5a78246fdd6
BLAKE2b-256 checksum
How to use checksums
b235b4d4f0f84fd5409a6c2f36261d2f8e10480efa6a02a0639a924b1d46d937
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

Security for pytest-agent-behavior 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pytest_agent_behavior-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page