AgentIdem

Reliability testing for state-changing AI agents.

Agents retry. Side effects do not rewind.

AgentIdem tests whether an agent behaves safely when execution is retried, interrupted, duplicated, or partially observed.

Installation

Requires Python 3.11 or newer.

pip install agentidem

Verify the CLI:

agentidem version

Expected output:

v1.0.0

CLI

AgentIdem v1 provides four commands:

agentidem test
agentidem record
agentidem replay
agentidem version

agentidem test

Test an agent for retry and side effect safety.

agentidem test module.path:function

Example:

agentidem test examples.cli_agents:unsafe_agent

AgentIdem automatically detects synchronous and asynchronous targets.

JSON output

agentidem test examples.cli_agents:unsafe_agent --json

Save a report

agentidem test examples.cli_agents:unsafe_agent --out report.json

You can use both together:

agentidem test examples.cli_agents:unsafe_agent \
  --json \
  --out report.json

Exit codes

agentidem record

Run an agent once and save its execution trace.

agentidem record module.path:function --out trace.json

Example:

agentidem record examples.cli_agents:unsafe_agent --out trace.json

The trace includes the target, operations, arguments, results, errors, statuses, and observation state.

If the target fails, AgentIdem still writes the partial trace and exits with code 2.

agentidem replay

Re-run a recorded target and compare it with the saved trace.

agentidem replay trace.json

Matching replay:

REPLAY MATCH: examples.cli_agents:unsafe_agent

Mismatching replay:

REPLAY MISMATCH: examples.cli_agents:unsafe_agent

Exit codes

Replay ignores nondeterministic trace IDs and timestamps.

agentidem version

Show the AgentIdem CLI version.

agentidem version

Output:

v1.0.0

Target format

CLI targets use:

module.path:function

Example:

my-project/
├── agents/
│   ├── __init__.py
│   └── refund.py
└── ...

With:

# agents/refund.py

def run() -> None:
    ...

Run:

agentidem test agents.refund:run

For v1, CLI targets should be zero-argument functions.

If your agent requires arguments, create a small wrapper:

from myapp.agent import process_order


def run() -> str:
    return process_order(
        order_id="ord_42",
        amount=100,
    )

Then run:

agentidem test myapp.agent_test:run

Development

Install development dependencies:

pip install -e ".[dev]"

Run the test suite:

pytest -v

License

Licensed under Apache License 2.0. See LICENSE.