AgentGauntlet

Every agent framework ships with a false sense of safety: pass the eval once, ship it, forget it. Real agents run in a world where context gets dropped, tools time out, and APIs return garbage — continuously, not once. AgentGauntlet treats that as the default assumption, not an edge case, and gives you two ways to watch it happen to your own agent.

Two ways to run it

In-process (Python agents on requests/httpx) — LangGraph, CrewAI, AutoGen, or any custom Python agent that calls the OpenAI/Anthropic SDKs:

import agentgauntlet
agentgauntlet.init(probability=0.1, frameworks=["openai", "requests"])

Two lines, before you build the graph/crew/agent. No other code changes.

Proxy (any agent, any language) — OpenClaw, Hermes Agent, or literally anything with an OPENAI_BASE_URL / ANTHROPIC_BASE_URL override, which is most agent runtimes regardless of what they're written in:

agentgauntlet proxy --upstream https://api.openai.com --port 8888
export OPENAI_BASE_URL=http://localhost:8888/v1
your-agent-here   # unmodified, any language

The proxy is the honest way to make good on "run this against whatever agent you have" — the in-process patch only ever reaches Python code in the same interpreter, so a Node.js or Go agent is invisible to it no matter how the hooks are written.

Injectors

In proxy mode all four compete in a single pool on every call, since everything reaching the proxy is presumed to be an LLM API call. In-process mode splits them: Amnesia/Distractor target LLM calls, Gaslighter/Mutator target tool calls. Either way, one intercepted call triggers at most one event — the blast radius is one weighted random draw per call, not an independent roll per injector.

Measuring what actually matters: task success, not process survival

with agentgauntlet.run():
    result = my_agent(user_query)
    if result.balance == expected_balance:
        agentgauntlet.mark_success()
    else:
        agentgauntlet.mark_failure("wrong balance reported")

If you never call mark_success/mark_failure, AgentGauntlet falls back to crash-detection and labels the score (unverified) — a much weaker signal, since an agent can process corrupted data, produce a wrong answer, and exit cleanly. examples/basic_agent.py --chaos reproduces exactly that: the Mutator corrupts an account balance, the agent reports the wrong number, nothing crashes, and the scorecard still shows a failure because the harness checks the actual answer.

🔥 CHAOS EVENT: MUTATOR 🧬 corrupted payload <- http://127.0.0.1:8931/tools/fetch_balance
...
        AgentGauntlet Post-Mortem Scorecard
┌────────────────────────────────┬──────────────────────────────────────────┐
│ Total Chaos Events Injected     │ 3                                        │
│ Task Outcome                    │ ❌ failure (reported 12505.0 vs 1250.5)   │
│ Resilience Score                │ 0%                                       │
└────────────────────────────────┴──────────────────────────────────────────┘

Configuration

agentgauntlet.init(
    probability=0.1,                      # shorthand: same weight for all four injectors
    blast_radius={"amnesia": 0.1, "distractor": 0.05, "gaslighter": 0.15, "mutator": 0.1},
    frameworks=["openai", "requests"],    # "httpx" also covers httpx.AsyncClient
    injectors=["amnesia", "gaslighter"],  # optional allow-list, default = all four
    targets=["api.mytools.com"],          # optional: restrict Gaslighter/Mutator to matching URLs
    seed=7,                               # reproducible runs
    timeout_range=(0, 30),                # Gaslighter's simulated timeout sleep, in seconds
    amnesia_strategy="random",            # or "oldest_first" for deterministic degradation
    mutation_rates={"boolean_flip": 0.5, "numeric_shift": 0.5, "key_mangle": 0.2},
)

agentgauntlet proxy --upstream https://api.openai.com --port 8888 \
    --probability 0.15 --seed 7 --timeout-min 0 --timeout-max 30

Values above ~0.25 on --probability mean chaos on essentially every call through the proxy (all four injectors share one pool there) — a legitimate "maximum chaos" demo setting, not a bug, but worth knowing before you crank it to 1.0 for a screenshot and wonder why every single call got hit.

How classification works, and its limits

A request body shaped like {"messages": [...]} or containing a system key is treated as an LLM call; anything else is a tool call, restricted to targets if you set one. This is a payload-shape heuristic, not real provider/client identification — it can misclassify an unrelated API that happens to send a messages field.

The proxy mode has its own, different limit: it only sees traffic an agent is explicitly configured to send through it (the base_url override pattern). It does not do transparent HTTPS interception via a system HTTP_PROXY

• generated CA certificate — that would require installing a locally-trusted root cert before anything works, which is exactly the setup friction that kills a "point it and go" first impression. Real, stated limitation, not a hidden one.

Known limitations (being upfront about scope)

• In-process mode is "any Python code on requests/httpx," not literally arbitrary tool mechanisms — subprocess tools, database drivers, and browser automation aren't touched, since there's no HTTP call to intercept in-process.
• OpenClaw and Hermes Agent are standalone runtimes, not Python libraries — the in-process patch cannot reach them at all; only proxy mode does, and only for their LLM API traffic specifically, not their internal tool/skill execution.
• No integration tests against LangGraph/CrewAI/AutoGen/OpenClaw/Hermes yet — this release has unit tests for the injectors and blast-radius logic (tests/) and a hand-run demo against a local mock server (examples/), not automated tests against real third-party frameworks.
• Crash attribution (sys.excepthook) blames whichever chaos event fired most recently before an unhandled exception — a heuristic, not a causal trace. agentgauntlet.run() + mark_success/mark_failure avoids needing it when the caller can check task correctness directly.

See examples/basic_agent.py (a real two-step tool-calling loop against a local mock LLM server, examples/mock_llm_server.py) for a runnable before/after demo, and tests/test_injectors.py for the injector and blast-radius unit tests.