AgentChaos

Evaluate agent system robustness through controlled, runtime, non-intrusive LLM API fault injection.

Overview

LLM-based agent systems issue multiple API calls per task, and each call can fail (HTTP 5xx, truncation, empty response, encoding corruption, schema violation). Once a faulty response occurs, it propagates through downstream agents and causes task failure. AgentChaos injects controlled faults at the HTTP transport layer — without modifying any agent source code — to evaluate robustness before these failures happen in production.

Quick Start

pip install agentchaos-sdk

import agentchaos

# Inject fault (your agent code needs ZERO changes)
agentchaos.inject("llm_error_single")
result = await my_agent(query)         # agent runs normally, unaware
agentchaos.disable()                   # stop
agentchaos.save_trace("trace.json")   # save full LLM call trace

# examples
git clone https://github.com/floritange/AgentChaos.git
cd AgentChaos
uv sync
uv run python examples/list_faults.py       # list all 65 faults
uv run python examples/agent_openai.py      # OpenAI agent: normal vs faulted
uv run python examples/agent_langchain.py   # LangChain agent
uv run python examples/agent_adk.py         # Google ADK agent
uv run python examples/eval_batch.py        # batch evaluation

How It Works

An HTTP-layer injection mechanism patches the HTTP client at runtime to intercept and modify LLM API responses according to the fault configuration, requiring no changes to any agent system.

Properties:

• Works with any framework using OpenAI-compatible APIs (OpenAI, LangChain, ADK, AutoGen, CrewAI, LiteLLM)
• Zero code changes — just inject() / disable() around your existing code
• Records full execution trace (raw input/output, token usage, timing) for every LLM call
• 65 pre-built fault configurations covering all real-world failure modes

API

import agentchaos

# Trace only (no fault)
agentchaos.inject(None)
result = await my_agent(query)
agentchaos.disable()
agentchaos.save_trace("trace_normal.json")

# Inject fault + trace
agentchaos.inject("llm_error_single")
result = await my_agent(query)
agentchaos.disable()
agentchaos.save_trace("trace_faulted.json")

# Batch evaluation
report = await agentchaos.eval(my_agent, query, faults="all")
print(report.summary())

Trace Format

{
  "call_index": 0,
  "raw_input": {"model": "gpt-5.5", "messages": [...], "tools": [...]},
  "raw_output": {
    "content": "The answer is 42.",
    "tool_calls": [],
    "finish_reason": "stop",
    "usage": {"prompt_tokens": 306, "completion_tokens": 54, "total_tokens": 360},
    "http_status": 200
  },
  "injected_output": {
    "content": "[API ERROR] HTTP 500: Internal Server Error.",
    "tool_calls": []
  },
  "timing": {"llm_latency_ms": 1523.4, "total_ms": 1524.1},
  "fault_applied": true
}

raw_output = LLM original response. injected_output = what the agent actually receives (only present when fault_applied: true).

Fault Taxonomy

We define a fault taxonomy by adapting the classical fault classification from distributed systems (Avizienis et al., 2004) to LLM API responses. The taxonomy covers crash, omission, and value faults on both content and tool call fields.

From Crash to Value, faults become progressively harder to detect. Crash faults produce obvious error signals and are typically retried. Value faults look like valid output and propagate silently — making them the most dangerous in practice.

65 = (6 fault types x 2 targets x 4 strategies) + 8 compound + 9 positional

Detailed documentation: docs/faults.md

Evaluation Results

Experimental Setup

Backbone LLMs: Claude-Sonnet-4.5, GPT-5.2, DeepSeek-V3.2, Seed-1.8

Metric: Δpass@1 = pass@1 (w/o fault) − pass@1 (w/ fault). Higher = more vulnerable.

RQ1: Overall Robustness Degradation (Claude-Sonnet-4.5)

Mini-SE is evaluated only on SWE-bench Pro (Δpass@1 = 0.87%).

RQ2: Impact of Fault Configurations

• Content faults cause higher Δpass@1 than tool call faults; only corrupt stays below 7%
• Persistent injection causes the highest Δpass@1 — up to 62.39% (MapCoder)
• Pipeline systems are most position-sensitive — single early fault drops pass@1 by up to 83.87%
• Compound content faults amplify degradation — up to 86.36% (MapCoder)

RQ3: Fault Diagnosis

Existing methods achieve below 53% accuracy on fault type and below 56% on fault step. Truncation — the most harmful fault — is identified with only 4.3% accuracy.

Key Findings

Documentation

• Fault Reference — Complete reference for all 65 fault configurations
• Examples — Runnable demos for OpenAI, LangChain, ADK

Citation

If you use AgentChaos in your research, please cite:

@article{agentchaos2026,
  title={AgentChaos: Chaos Engineering for Robust Agent Evaluation via LLM API Fault Injection},
  year={2026}
}

License

MIT -- see LICENSE.