Skip to main content

AgentChaos

Open-source reliability testing for tool-using AI agents. Record agent runs, profile cost, and catch operational regressions in CI — before they hit production.

0.1.0 — v0 wedge: detect a cost/latency/tool-call regression in CI. See PROGRESS.md and the roadmap.

The problem

Tool-using agents fail in production in ways quality-focused eval tools don't catch:

  • Cost silently doubles after a prompt or model change.
  • A flaky tool returns 503 and the agent retries 12 times, racking up $4 in model calls.
  • Retrieval returns 12 chunks instead of 5, blowing the planner's context budget.
  • The "fallback when a tool fails" path was never tested, because tools never fail in dev.

Eval tools (LangSmith, Braintrust, DeepEval) score answer quality. Load tools (k6, Locust) don't understand the agent loop. AgentChaos covers the operational gap: deliberate, reproducible regression detection for tool-using agents, runnable in CI.

Quickstart

pip install agentchaos-reliability
agentchaos init my-agent-tests
cd my-agent-tests
agentchaos doctor scenarios/example.yaml      # validate + ping your agent
agentchaos run scenarios/example.yaml          # record a baseline trace
# ... change your agent (prompt, model, retrieval) ...
agentchaos run scenarios/example.yaml --baseline runs/baseline.jsonl

A scenario is plain YAML: a conversation, the tools you expect, and operational budgets.

id: refund-agent
agent:
  type: http
  endpoint: http://127.0.0.1:8080/chat
conversation:
  - user: "I want to return my order."
  - user: "My order number is 12345."
expect:
  must_call_tools: [get_order, create_return_label]
  final_response_contains: [return label]
budgets:
  max_cost_usd: 0.05
  max_cost_regression_pct: 20          # fail if cost grows >20% vs baseline
  max_input_token_regression_pct: 30

What you get

Re-run after a change and AgentChaos diffs against the baseline, gates on your budgets, and tells you why cost moved — then exits non-zero so CI fails:

AgentChaos — refund-agent

Verdict: FAIL

Why:
  - [regression_budget] max_cost_regression_pct: cost regressed +68.2% (limit +20.0%)
  - [regression_budget] max_input_token_regression_pct: input_tokens regressed +90.8% (limit +30.0%)

Metrics:
  Metric                      Baseline       Current         Δ  Status
  --------------------------------------------------------------------
  Cost                         $0.0004       $0.0006    +68.2%  FAIL
  Input tokens                    1850          3530    +90.8%  FAIL
  Output tokens                    150           150     +0.0%  PASS
  LLM calls                          3             3     +0.0%  PASS
  Tool calls                         2             2     +0.0%  PASS

Tool sequence:
  baseline: get_order -> create_return_label
  current:  get_order -> create_return_label

Possible contributors:
  - input_tokens grew +90.8% (+1680 tokens)
  - per-model cost grew on gpt-4o-mini: $0.000368 → $0.000619 (+68.2%)
  - metadata.rag_chunks changed on model_call planner: 5 → 12  [correlates]

Exit code: 2

The cause detection is observation-first: it leads with deltas and lists possible contributors with confidence (observed / correlates / computed). Only a model swap claims a hard dollar contribution.

Try it now — the refund-agent demo reproduces the report above in under a minute.

How it works

scenario.yaml → run → trace.jsonl → metrics → compare(baseline) → terminal verdict + exit code

Your agent is any HTTP endpoint that takes a message and returns a response. AgentChaos auto-detects one of three fidelity tiers from your response shape — full per-call cost attribution, aggregate usage, or message-only — and tells you which on the first run. No SDK, no instrumentation, no framework lock-in.

See the HTTP integration contract for the exact request/response shapes and fidelity tiers.

Command Purpose
agentchaos init Scaffold a scenarios/ + runs/ folder
agentchaos doctor Validate a scenario and probe the endpoint
agentchaos run Execute a scenario, record a trace, optionally diff a baseline
agentchaos compare Diff two existing traces — pure analysis, no agent calls

Exit codes: 0 pass · 1 config error · 2 budget/expectation violation · 4 endpoint unreachable.

What it is not

  • Not a generic LLM eval platform. Use DeepEval, Braintrust, or LangSmith for answer-quality scoring.
  • Not a prompt management tool. Use Langfuse or git.
  • Not an observability dashboard. Use Phoenix or Datadog.
  • Not a load testing framework. Use k6 or Locust.

AgentChaos does one thing: catch operational failures and cost regressions in tool-using agents, in CI.

Roadmap

  • v0 (now) — cost/regression profiling with baseline diff. ✅
  • v1 — chaos injection (tool failure/latency), loop & retry-storm detection, record/replay, OTel emission, GitHub Action.
  • v2 — framework adapters (LangGraph, OpenAI Agents SDK), MCP proxy + contract tests.

License

Apache 2.0.

Metadata

Release files for agentchaos-reliability 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agentchaos-reliability 0.1.0
File Size Uploaded Provenance Actions
agentchaos_reliability-0.1.0.tar.gz 109.5 kB Without provenance Details for agentchaos_reliability-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agentchaos-reliability 0.1.0
File Interpreter ABI Platform Actions
agentchaos_reliability-0.1.0-py3-none-any.whl Python 3 none any Details for agentchaos_reliability-0.1.0-py3-none-any.whl

Total release size: 142.5 kB

Release files / agentchaos_reliability-0.1.0.tar.gz

Download URL agentchaos_reliability-0.1.0.tar.gz
Size 109.5 kB
Tags Source
SHA-256 checksum
How to use checksums
de529113b11a80c6aa470e96631ccd6b218e9a0a2efe5463b70edb9e94833b41
BLAKE2b-256 checksum
How to use checksums
c1125c64eca2a4519a196c25c3225cf63398f052eb4fdd58ba92a1a368c337d6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.2 {"installer":{"name":"uv","version":"0.11.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / agentchaos_reliability-0.1.0-py3-none-any.whl

Download URL agentchaos_reliability-0.1.0-py3-none-any.whl
Size 33.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2fdc3680a7e83e56ae8eb39690905b0d2bfced3a52bd8bd88f310b4a76e2d0bd
BLAKE2b-256 checksum
How to use checksums
8361c3e1ebbee9cb6c3a7986afebca702dc42e79de59b02fd280118a2e7e21d8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.2 {"installer":{"name":"uv","version":"0.11.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for agentchaos-reliability 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page