Skip to main content

AgentPostmortem

Find out how your AI agent fails before your customers do.

PyPI CI License: Apache-2.0 Python 3.11 to 3.13

AgentPostmortem is an open reliability benchmark and fault-injection runner for AI agents. It breaks an agent's world on purpose, safely, and records what the agent actually did.

tool_call refund(ch_0001, 20.00)  -> timeout (the refund went through, the response was lost)
retry     refund(ch_0001, 20.00)  -> ok
ledger:   payments.refund(ch_0001) applied x2        FAIL no_duplicate_effects

Most agent failures that cost money are not the model saying something wrong. They are systems failures around the model: a retry that charges twice, a crash that repeats finished work, an error that turns into "Done!", a tool answer that is stale or wrong. AgentPostmortem reproduces those incidents as deterministic scenarios and judges every run by its side effects, not by how the transcript reads.

Quick start

pip install agentpostmortem-bench      # installs the `apm` command and `import agentpostmortem`

apm list                                  # 25 scenarios
apm show APM-S001                         # the incident, the plan, the faults, the checks
apm run --scenario all --agent all        # writes apm-results/results.json and report.html

Framework mode needs no API key and costs nothing: the model is replaced by a scripted one, so every run is exactly reproducible and only the harness around the model is under test.

What a run tells you

Five reference agents implement the harness policies teams actually ship. Same plan, same faults, different safety nets:

Agent Policy Scenarios passed (of 25)
naive one attempt, no retries 4
retrying retries on error, no idempotency keys 1
idempotent retries with a stable idempotency key per step 6
durable idempotent, plus checkpoints that survive a crash 9
careful durable, plus response validation, backoff and honest failure reports 23

Adding retries without idempotency keys makes an agent worse, not better. Durability fixes crashes but not lying tools. The two scenarios careful still fails are documented as needing a check no generic harness can provide.

How it works

scenario  ->  run  ->  ledger  ->  verdicts  ->  report
  • Scenarios are YAML files. Each reproduces a real incident pattern: the task, the simulated services, which fault fires and when, and what correct behaviour looks like.
  • Simulated services (payments, email, SMS, helpdesk, inventory, orders, a key-value store, a model API, an event feed) are deterministic and honour real semantics such as idempotency keys and conditional writes.
  • The side-effect ledger records every real-world effect: what was applied, what was deduplicated, and when.
  • Oracles turn the ledger and the agent's final claim into verdicts with evidence: no_duplicate_effects, honest_completion, resumed_without_repeat, validated_before_use, retries_respect_backoff, cost_within_budget and more.
  • Validity. A run only counts if the faults and crash it declares actually happened. If an agent gives up before a fault can fire, the run is reported as invalid, never as a pass.

Fault catalogue

Fault What it simulates
timeout No answer in time; nothing happened
timeout_after_success The work happened, the response was lost
server_error A 5xx; nothing happened
rate_limit A 429 with retry-after, or a fixed-window quota
malformed Missing fields or wrong types
stale An older, well formed value (or a redelivered event)
lying A fast, well formed, wrong answer
partial A batch call that succeeds as a whole while some items fail
crash The agent process dies after step N and restarts

Scenarios cover six categories: duplicate effects (5), honest completion (4), progress and resume (4), bad data (5), runaway cost (4) and compound failures (3).

Test real frameworks

pip install "agentpostmortem-bench[frameworks]"
apm run --scenario all --agent langgraph langgraph-checkpointer openai-agents openai-agents-session

Adapters drive each framework's own agent loop, tool wrappers and error handling with a scripted model. They add no retries, keys or validation of their own, so what you measure is the framework's default behaviour and its documented options. Available today: LangGraph (create_agent with defaults, a checkpointer, tool-retry middleware and tool-error middleware) and the OpenAI Agents SDK (defaults, raise-on-error and SQLite sessions).

Framework results are published only after the findings have been shared with each framework's maintainers.

Test any agent over MCP

pip install "agentpostmortem-bench[mcp]"
claude mcp add apm -- apm serve --scenario APM-S001 --agent-name claude-code

apm serve exposes one scenario's simulated services as tools over MCP (stdio) or plain HTTP on 127.0.0.1, so Claude Code, Cursor, Codex CLI or an agent in any language can be tested. The agent ends with a finish tool; the same ledger, oracles and validity rules produce the same results.json and report.html as apm run. Crash scenarios are reported invalid for now, because the harness cannot kill an external process. See docs/mcp.md.

Use it in CI

- run: pip install "agentpostmortem-bench"
- run: apm run --scenario all --agent careful --fail-on-fail

apm run exits 0 when every run completed, whatever the verdicts. --fail-on-fail exits 1 if any run failed. Results are written as results.json (schema version 1) and a self-contained report.html that works offline, in light and dark mode.

Hermes Agent

integrations/hermes ships a Hermes Agent skill that runs the benchmark and explains the results in chat, plus instructions for sending result alerts through a Hermes webhook route.

Write a scenario

id: APM-S002
title: Customer charged twice after a lost response
category: duplicate_effects
incident:
  pattern: Charge succeeds, response lost, client retries
  summary: The charge succeeds but the response never arrives. Retrying without an idempotency key creates a second charge.
task: Charge customer cus_lin 12.99 for order 1042.
services: [payments]
plan:
  - {id: charge, service: payments, operation: charge, args: {customer: cus_lin, amount: 1299}}
faults:
  - {service: payments, operation: charge, kind: timeout_after_success, trigger: {calls: [1]}}
oracles:
  - {name: no_duplicate_effects}
  - {name: effect_completed, params: {service: payments, operation: charge, count: 1}}

See CONTRIBUTING.md for how to add one and pin its expected verdicts.

Verdicts come from side effects in deterministic simulated services, and runs where a declared fault never fired are excluded. This builds on a growing body of work, and we do not claim to be first:

  • LIMBO, "Where Does Exactly-Once Live?" (arXiv 2609.29095): deterministic services, service-boundary faults and a ledger of committed effects.
  • AgentChaos (arXiv 2608.06790, ASE 2026): fault injection at the LLM API layer; robustness depends on system implementation more than the model.
  • ReliabilityBench (arXiv 2601.06112): reliability under faults and perturbations with end-state checks.
  • RetryBench (arXiv 2610.15884) and Fabrication After Tool Failure (arXiv 2609.14758): duplicate and omission accounting, and false success after tool errors.
  • SSCBench (arXiv 2610.11514): why an injected fault is not automatically a tested one, which is the reason runs report validity.

What AgentPostmortem adds is a maintained, reproducible benchmark of agent frameworks and harness setups, with a CLI for CI and a report people can read.

Status

Alpha. The scenario format and the results schema may change before 1.0; results.json carries a schema_version so tools can tell.

Questions, findings or a scenario idea: open an issue, or email work@royalpinto007.com. Security reports: see SECURITY.md.

License

Apache-2.0

Metadata

Release files for agentpostmortem-bench 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agentpostmortem-bench 0.1.0
File Size Uploaded Provenance Actions
agentpostmortem_bench-0.1.0.tar.gz 119.0 kB Without provenance Details for agentpostmortem_bench-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agentpostmortem-bench 0.1.0
File Interpreter ABI Platform Actions
agentpostmortem_bench-0.1.0-py3-none-any.whl Python 3 none any Details for agentpostmortem_bench-0.1.0-py3-none-any.whl

Total release size: 241.4 kB

Release files / agentpostmortem_bench-0.1.0.tar.gz

Download URL agentpostmortem_bench-0.1.0.tar.gz
Size 119.0 kB
Tags Source
SHA-256 checksum
How to use checksums
ef7bd0a7c73d6eb18e7fc8adc93db964a57de1e54e7e7a865817ef6cca0418fd
BLAKE2b-256 checksum
How to use checksums
720e6fe89202a086aa1714e91c7f4ef6198486a5b2eb1f5e13dbe2fcd92e42a8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.17 {"installer":{"name":"uv","version":"0.9.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / agentpostmortem_bench-0.1.0-py3-none-any.whl

Download URL agentpostmortem_bench-0.1.0-py3-none-any.whl
Size 122.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a4db46ded491fdf8f6c7ca653b1b80d5ffb17469b21804d1bc1bbc562b5d259c
BLAKE2b-256 checksum
How to use checksums
5dace5815c40d746539d916e2cd740194163de91a45ce1a6c2e8a23ba965ca00
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.17 {"installer":{"name":"uv","version":"0.9.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for agentpostmortem-bench 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page