Swarmbreaker

Load- and chaos-testing for AI agent orchestration plumbing — transport, concurrency, resource limits, retry/repair paths, context budgeting. Swarmbreaker drives a population of heterogeneous synthetic agents against a large fabric of mock MCP tools, injects the statistical pathology of real model output at the wire, and reports where the plumbing breaks and how (knee points with confidence intervals, labeled by epistemic confidence).

It is deliberately not a model-quality benchmark. The mock is the measurement instrument; a real LLM is only the calibration standard. Read SPEC.md — the design specification is the source of truth for everything here.

A real recording, real time: scripts/render_demo_svg.py replays an actual swarmbreaker demo run byte-for-byte, so this image cannot show output the tool didn't produce.

Try it in 30 seconds

uvx --from git+https://github.com/wczaja/swarmbreaker swarmbreaker demo

No API keys, no Docker, no config. Two acts over the real pipeline: a compressed null-SUT self-test measures the rig ceiling (SPEC §6.11 — an instrument that can't pass its own polite baseline refuses to make claims), then a chaos scenario ramps open-loop load through a §7 injector mix until an SLI knees — and it ends with the finding, labeled by epistemic confidence and validity-guarded against that ceiling. A weekly public CI run publishes the same output in its job summary, so you can inspect a fresh run without installing anything.

The chaos is declarative, and the demo hands you its own config to tinker with — this scenario produced the finding on the right:

run:  { seed: 1729, duration_s: 15, time_scale: 0.05 }
scale: { agents: 16, tools: 128 }
load:
  arrival: { process: poisson, rate_per_s: 12 }
population:
  - { name: well_behaved, weight: 0.7 }
  - { name: chatty_high_fanout, weight: 0.2 }
  - { name: poison, weight: 0.1 }
brain:
  injectors:
    invalid_json_args:   { rate: 0.04 }
    near_miss_tool_name: { rate: 0.04 }
    truncated_stream:    { rate: 0.02 }
    repair_storm:        { rate: 0.01, burst: 6 }
scenario:
  - graduated_ramp: { shape: exponential, start: 0.08 }
  - rate_limit_cascade: { rps: 6, burst: 6 }

finding  tool_err_rate knees at 3.3 tasks/s
         (95% CI 2.7–4.7 · two estimators agree)
         [projected × scripted]

         the 429 cascade — offered load crossed
         the shared 6 rps hot-tool bucket

         validity PASS — rig at 35% of its own
         ceiling; this behavior is the SUT's,
         not the rig's

Swap the failure mode with --scenario slow-loris (hot tools hang with no server-side timeout) or --scenario herd (barrier-released burst); edit the YAML and re-run it with swarmbreaker run to go past the demo. swarmbreaker demo --live first fits the injector rates, latency, and repair policy to a real model via the Phase-4 calibration loop (local Ollama by default, or any OpenAI-compatible endpoint with one key) — the same finding then honestly labels calibrated instead of scripted.

Status

Pre-release. The SPEC.md §14 build sequence is complete through v1.x:

• Phase 0 — skeleton + rig self-test
• Phase 1 — injector pipeline (malformation taxonomy §7)
• Phase 2 — scenario engine + chaos catalog (§8)
• Phase 3 — telemetry, knee estimation, confidence labeling, reporting (v1 release line)
• Phase 4 — calibration loop (v1.x)
• Phase 5 — integration & docs (v1.x): docket handoff, graph-pattern-hardened LangGraph adapter, Pydantic AI + OpenAI Agents SDK thin adapters (§16.4), scaling guide

Beyond the demo

uv venv && . .venv/bin/activate
uv pip install -e ".[dev]"

swarmbreaker demo                            # the 30-second tour, from source
swarmbreaker selftest --preset laptop --out runs/selftest    # rig ceiling at laptop scale (§6.11)
swarmbreaker run examples/quickstart.yaml --out runs/quickstart   # a 60s knee-hunting sweep
open runs/quickstart/report.html

Every run directory contains findings.json (machine-readable, SPEC §11), report.html (SLI-vs-load curves with knee CIs), events-*.jsonl (the raw corpus), and validity.json (whether the rig had headroom — findings from a saturated rig are refused).

Calibration (Phase 4)

Findings are labeled scripted until the injector parameters are fitted against a real model (SPEC §11). Point the harness at any OpenAI-compatible upstream — a local model (Tier 2: Ollama, llama.cpp, vLLM) or a hosted API (Tier 3):

# Tier 2 (local, free): discover + fit against e.g. Ollama
swarmbreaker calibrate --base-url http://localhost:11434/v1 --model llama3.2 \
    --tier local --sessions 60 --out calibration/llama3.2.json

# Tier 3 (live API): same command, your endpoint + OPENAI_API_KEY
swarmbreaker calibrate --base-url https://api.openai.com/v1 --model gpt-4.1-mini \
    --tier live --sessions 60 --out calibration/gpt41mini.json

Then reference it from a run config (brain: { calibration: calibration/llama3.2.json }). Rates are fitted per §7 family, stratified by context length × schema complexity; TTFT/token-rate and the repair policy are fitted too. Findings then carry per-family "calibrated-against" stamps and flip to calibrated — until the stamps expire (default 90 days) or you hand-override a calibrated rate, at which point they honestly decay back to scripted. Discovery runs (brain: { tier: local }) proxy the swarm through the real model and write discovery.json — the shopping list for new scripted injectors.

Other useful commands: swarmbreaker advise (which failure modes your hardware can reproduce, §6.7 — start with the scaling guide), swarmbreaker baseline + swarmbreaker gate (statistical regression gating, §12 — this repo's own CI uses them), and swarmbreaker corpus (validate/push the docket triage corpus, docs/docket-handoff.md).

SUTs

The reference SUT is an unmodified LangGraph agent (adapter: langgraph; graph patterns react, supervisor, plan_execute). Thin adapters for Pydantic AI (adapter: pydantic_ai) and the OpenAI Agents SDK (adapter: openai_agents) prove the core is orchestrator-agnostic (SPEC §16.4); each is an optional extra (pip install "swarmbreaker[langgraph]", [pydantic-ai], [openai-agents]). adapter: null_sut is the instrument's own baseline (§6.11).

Contributing

See CONTRIBUTING.md — the short version: SPEC.md is the source of truth, phase exits are executable tests, and all stimulus randomness must flow through seeded substreams.

License

Apache-2.0 — see LICENSE.