Mayhem

Chaos engineering for Docker & Podman (Kubernetes on the roadmap): discover your system, break it on purpose, prove it recovers — every time, with evidence.

Mayhem is a drill engine, not a command library. It turns your docker-compose blueprint into a live topology graph, compiles one declarative kind: drill YAML file into a frozen, safety-gated execution plan, injects faults through a capability-aware toolkit, and ends every run with a machine verdict derived from the observations it actually recorded. Steps, probes, criteria evaluations, and decisions all land in SQLite — nothing is "trust me, it worked."

compose blueprint ─▶ topology graph ─▶ compile drill spec ─▶ frozen plan
                                                              │
                        verdict ◀── machine criteria ◀── inject → observe → recover
                                                              │
                                                SQLite evidence + decision trace

Contents

• Installation
• Why Mayhem
• What a Run Looks Like
• Quickstart
• Authoring a Drill
• CLI Reference
• Configuration
• Safety Model
• Exit Codes
• Architecture
• Development

---

Installation

Mayhem publishes to PyPI as mayhem-cli — the bare mayhem name is taken, so you install the CLI under that name. The console commands it puts on your PATH are still mayhem (and mayhem-agent).

Requires Python 3.12+.

pip install mayhem-cli

That is it — mayhem is now on your PATH:

mayhem --help

Isolated installs (recommended on macOS/Linux) — keeps the CLI out of your global Python:

pipx install mayhem-cli        # or: uv tool install mayhem-cli

Upgrading — each git tag (v0.5.0 or 0.5.0) publishes that exact version:

pip install -U mayhem-cli

Need the source checkout instead? See Development.

---

Why Mayhem

Most chaos tooling hands you a list of commands — "kill this container," "add latency here" — and leaves you to figure out the blast radius, the cleanup, and what "healthy" means. Mayhem treats a drill as a planned, gated, evidenced experiment instead.

	Hand-rolled chaos scripts	Mayhem
Targeting	You pick a container and hope	Faults plan against a discovered topology graph, so targets stay real
Safety	Your discipline	Risk ceiling, max_faults budget, and an impact gate that re-proves every fault before anything is injected
Definition	Throwaway shell one-liners	One kind: drill YAML: per-container faults, ordering, checks, success criteria, observability
Cleanup	You remember to	Every fault ships a compensation contract; the janitor sweeps dirty leases after crashes
Verdict	Eyeball the dashboards	PASS/FAIL from typed success criteria evaluated over recorded observations
Evidence	Shell history	Immutable SQLite journal: steps, probes, decisions, outcome

Concretely:

1. Topology-aware planning. Mayhem reads your compose blueprint and discovers the live services, hosts, and dependencies to build a graph. Faults are planned against that graph, not applied blind; compose project filtering keeps unrelated stacks out.

2. Drill specs, not scripts. One YAML file declares everything — faults, cross-container ordering, checks, success criteria, observability sources — and you define what "healthy" looks like before you break anything.

3. Machine verdicts. Optional typed success criteria turn every run into a PASS/FAIL verdict derived from the observations actually recorded — no eyeballing.

4. Automatic recovery. Every fault ships a compensation contract: an undo op plus a verification probe generated from the same plan parameters, so what was injected is exactly what gets removed and re-proven healthy. If a round fails or the controller crashes, the janitor sweeps dirty leases and reconciles the workload — no manual cleanup. The per-fault lifecycle is documented in docs/compensation.md.

5. Honest evidence. Runs persist an immutable event journal, criteria evaluations, and a governing-decision trace in SQLite. No "trust me, it worked."

6. Prefix-shortened CLI. Type mayhem r instead of mayhem run; mayhem e v means mayhem experiment validate. Every command abbreviates to any unique prefix.

---

What a Run Looks Like

A clean run needs no interpretation — the verdict is one line away. Truncated for readability:

$ mayhem run mayhem.yaml --compose docker-compose.yml

# Run r-process-drill-8f2a1c
**status**: completed
**verdict**: pass
**success criteria**: ALL PASS
- [PASS] status:api-up.status PASS (expected 200, got 200)
- [PASS] latency:api-up.latency_ms PASS (31.2 <= 500.0)
**observations**: 2/2 sources collected
**decisions**: 5 governing decision revisions (snapshot in the run row)
**wall**: 32.4s

run r-process-drill-8f2a1c — inspect with `mayhem history r-process-drill-8f2a1c`

Reading the transcript, top to bottom:

1. Status — the run's machine state: completed (or failed/aborted with a non-zero exit and dirty-lease warnings).
2. Verdict — pass/fail derived from the success criteria you declared in the spec; undecided when a run aborts or the spec has no success block.
3. Success criteria — every criterion evaluated against real observations, one line each, so a failure tells you exactly what drifted.
4. Observations — how many configured evidence sources actually delivered data (probes, logs, metrics …).
5. Decisions — the governing decision revisions that shaped this run; the decision trace is queryable afterward via mayhem history.
6. Copy-paste handle — the run id for the follow-up commands below.

From there: mayhem status lists recent runs, mayhem status --run <run-id> shows full recorded metadata, and mayhem history <run-id> replays the complete event journal (every step, probe sample, and lease for that run). Add --debug to mayhem run to stream each step live as it happens ([ok] injected proc.pause 10s into testcase-api, [ok] recovered ... (compensation ok)).

---

Quickstart

Prerequisites

• Python 3.12+
• Docker with Compose v2 (or Podman, used via the --podman flag)
• No host tooling required up front: mayhem toolkit list probes for the capabilities (docker, podman, network tooling, k6, …) each fault needs, and the impact gate refuses to run anything it cannot prove.

Install

pip install mayhem-cli

The PyPI package is mayhem-cli (mayhem is taken); the console command remains mayhem. Full instructions in Installation. For a source checkout use pip install -e . (see Development).

Run the bundled example

A complete, self-contained example lives in examples/testCase/ — a six-service compose stack (API, web, load balancer, dual download builders, and Postgres), its drill spec, and the fixtures it relies on.

# 0. Bring the stack up.
cd examples/testCase
docker compose up -d

# 1. See the topology Mayhem will plan against (blueprint ─▶ live graph).
mayhem topology discover --compose docker-compose.yml

# 2. Compile the spec and run every safety gate — injects nothing.
mayhem validate mayhem.yaml --compose docker-compose.yml

# 3. Print the frozen execution plan as JSON.
mayhem plan mayhem.yaml --compose docker-compose.yml

# 4. Execute the drill and print the run summary.
mayhem run mayhem.yaml --compose docker-compose.yml

# (Variant) limit a run to ONE container — the pinned container's faults run,
# every other container's rounds are dropped from the frozen plan.
mayhem run mayhem.yaml --compose docker-compose.yml --ctr testcase-api

# (Variant) `maniac` draws its random rounds against a single container too —
# handy when you want to chaos-test one container without touching the others.
mayhem maniac --compose docker-compose.yml --ctr testcase-api

# 5. Replay any run's evidence.
mayhem status --run <run-id>
mayhem history <run-id>

mayhem run prints a copy-paste run <run-id> — inspect with mayhem history <run-id> line at the end; that id is all you need for the evidence commands.

The example spec exercises 38 distinct faults (every catalog entry that applies to a Docker/Podman compose service) across the testcase-lb load-balancer — CPU/memory/fd/disk pressure, load spikes, protocol abuse, network latency/bandwidth/packet-loss, dependency and database and DNS faults, TLS failure, container kill/restart/pause, HTTP error injection — capped at one concurrent fault (max_faults: 1, risk_ceiling: critical) with auto-recovery off (recovery: false), so the downstream checks observe whether the stack self-heals on its own. The remaining nine k8s.* catalog faults are exercised against a Kubernetes blueprint in examples/k8s (planning-only until the M8 driver lands), so every fault in the catalog has an example.

Omit --compose and Mayhem auto-detects docker-compose.yml (or compose.yml) in the current directory.

validate and plan work against the blueprint alone and never touch live containers. run re-proves every fault at the impact gate against the live graph and bypasses — or, without --skip-gate, refuses — anything it cannot prove injectable.

---

Authoring a Drill

A drill is one kind: drill YAML file — the DSL reference lives in docs/drill-spec.md. The shape:

apiVersion: "mayhem/v1"
kind: drill
name: checkout-recovery
hypothesis: "checkout stays available while cart writes are throttled"

config:
  risk_ceiling: high      # refuse faults riskier than this (policy ceilings only tighten)
  max_faults: 1           # never inject more than one fault at once
  timeout: 30m

containers:
  cart-api:               # keys ARE compose container_name values
    faults:
      - fault: net.latency
        duration: 10s
        params: { seconds: 5s, jitter_ms: 10 }

execution:
  - parallel: [cart-api]
  - check_spec:           # locus-aware checks
      - id: cart-health
        probe: { type: http, url: http://cart-api:8080/_health, expected_status: 200 }
        execution: service
        target: cart-api

success:                  # machine verdict
  require_all: true
  criteria:
    - type: status
      source_id: cart-health.status
      expected: 200
    - type: latency
      source_id: cart-health.latency_ms
      lt_ms: 800

observability:            # evidence sources
  sources:
    - kind: logs
      source_id: cart-logs
      container: cart-api
      tail: 200
    - kind: probe
      source_id: cart-probe
      probe: { type: http, url: http://cart-api:8080/_health }
      cadence: 2s

Validate with mayhem validate mayhem.yaml; unknown parameters, out-of-range durations, untargetable node kinds, and capability gaps are all compile-time errors — before anything is injected.

Campaigns

A campaign groups drill specs under one execution umbrella (ADR-0022/0023): experiments run sequentially in priority order, and the campaign's failure policy and time window govern the whole run.

mayhem campaign create black-friday \
  --description "BFCM chaos" --hypothesis "checkout survives every single-fault failure"
mayhem campaign add-experiment black-friday mayhem.yaml
mayhem campaign add-experiment black-friday checkout-recovery.yaml
mayhem campaign start black-friday                      # draft -> running
mayhem campaign run black-friday --compose docker-compose.yml

A campaign is born draft and moves through scheduled → running → paused → completed / aborted (archive closes a finished campaign). Per-experiment results land in the observations table under the campaign id, and the failure policy selects the next action when one experiment fails:

on_experiment_failure	Meaning
abort_campaign (default)	Stop the remaining experiments.
skip_and_continue	Record the failure and run the next experiment.
retry_then_abort	Retry the failed experiment once, then abort the campaign.

Window fields (window_json): start_epoch_s / end_epoch_s, max_duration_s (hard stop), and cooldown_between_experiments_s between successive experiments.

---

CLI Reference

Global options (accepted at any level, before or after the command):

Option	Meaning
--db PATH	SQLite database path (default mayhem.db).
--config PATH	Path to mayhem.yaml (overrides auto-detection).
--profile NAME	Configuration profile to merge.
--allow-critical	Acknowledge critical-risk faults (e.g. k8s.node_drain).
--skip-gate	Run even when the impact gate proved some faults inert.
--podman	Use Podman instead of Docker.
--debug	Re-raise errors instead of rendering them.

Commands:

Command	Description
mayhem topology discover	Discover live services/hosts and the dependency edges from the blueprint.
mayhem validate SPEC	Compile a drill spec and run every safety gate without injecting.
mayhem plan SPEC	Compile against the topology and print the frozen plan JSON.
mayhem run SPEC	Compile and execute a drill; print the run summary. --ctr CONTAINER scopes execution to one container (any container_name: value or runtime container name from the blueprint).
mayhem maniac SPEC	Compile and execute a random-injection drill — draws run_level single-fault rounds governed by the maniac seed/level (spec config.maniac, falling back to the maniac: layer of mayhem.yaml). --ctr CONTAINER narrows every draw (and the zero-config synthesized pool) to one container.
mayhem status	Show runs recorded in the database (--json supported).
mayhem history RUN_ID	Replay steps, events, and leases recorded for one run.
mayhem recover RUN_ID	Recover every orphaned fault lease belonging to a run.
mayhem janitor	Sweep leases past their TTL; expire pending runs; compensate.
mayhem toolkit faults	List the fault catalog with risk and compensatability.
mayhem toolkit list	Probe the host for the tools/capabilities faults require.
mayhem experiment show SPEC	Print the parsed drill spec as JSON.
mayhem experiment validate	Alias of validate.
mayhem config show / mayhem config validate	Inspect / validate the effective layered configuration (alias cfg). show --json also reports each section's provenance.
mayhem campaign list	List campaigns (draft/scheduled/running/…; --json).
mayhem campaign create NAME	Create a draft campaign (--description, --hypothesis).
mayhem campaign show / status / delete ID	Inspect, poll, or delete a campaign.
mayhem campaign add-experiment ID SPEC	Append a drill spec file to a campaign.
mayhem campaign start / abort / archive ID	Move a campaign through its lifecycle.
mayhem campaign run ID	Execute every experiment sequentially against the compose topology, honoring the campaign policy and window (--no-gate bypasses the impact gate).

Every command (and the whole tree) abbreviates to any unique prefix: mayhem ex valid, mayhem t f.

---

Configuration

Runtime policies live in mayhem.yaml — the configuration file, distinct from a kind: drill spec — auto-detected in the cwd or given with --config. The effective view is one command away: mayhem config show (alias cfg) prints the resolved configuration and the provenance of every section; mayhem config validate refuses unknown keys, a missing or wrong apiVersion, and out-of-range sections before anything runs.

apiVersion: mayhem/v1        # required; anything else is rejected
policy:
  allow_faults: null         # null = whole catalog; set to restrict
  deny_faults: []            # fault ids never injectable
  risk_ceiling: null         # tightened by the drill ceiling at plan time
  allow_critical: false      # config-side half of the critical opt-in
blast_radius:
  max_services_pct: 50.0
  max_hosts: 2
  max_concurrent_faults: 3
  max_duration_per_fault_s: 300.0
  forbidden_fault_pairs: []  # e.g. ["net.packet_loss", "net.bandwidth"]
storage:
  path: mayhem.db            # SQLite database (same default as --db)
  artifacts_dir: .mayhem/artifacts
toolkit:
  binaries: {}               # pin a named tool's binary, keyed by fault backend
runtime: docker              # docker | podman (CLI: --podman)
target:
  containers: []             # explicit targets when no compose file is used
log_level: INFO              # DEBUG | INFO | WARNING | ERROR
maniac:                      # fallback for `mayhem maniac` when the spec omits config.maniac
  level: 2
  run_level: 10
  seed: null                 # null = fresh random seed each run

Layering, in increasing precedence: built-in defaults → mayhem.yaml → mayhem.{profile}.yaml → environment variables → CLI flags. Profile overlays are separate per-profile files (selected with --profile NAME) — there is no profiles: key inside mayhem.yaml. The environment layer only honours allowlisted variables: MAYHEM_STORAGE_PATH, MAYHEM_ARTIFACTS_DIR, MAYHEM_LOG_LEVEL.

mayhem config show            # YAML + sourced-from comments
mayhem config show --json     # {"config": …, "sources": …}
mayhem config validate        # exit 0 / 3 on invalid layers

Drill-level config.risk_ceiling composes with the policy ceiling and can only tighten it.

---

Safety Model

• Risk ceilings. Every catalog fault carries a risk level; injection is refused when either the policy or the drill ceiling is exceeded. --allow-critical is the operator-side acknowledgment.
• Concurrency budget. max_faults caps simultaneously-injected faults; a wider parallel: step queues into rounds.
• Duration caps. Per-fault duration beyond the catalog maximum is a compile error.
• Capability gating. Faults declare the capabilities they need (docker engine, net_admin, process control, …); the plan is proven against the live graph by the impact gate before run — never assumed.
• Compensation contracts. Reversible faults run their declared inverse; irreversible ones are followed by workload reconciliation. A failed round aborts-and-recovers its own faults first, then propagates.
• Lease hygiene. Fault rounds hold leases with a TTL; mayhem janitor sweeps orphaned leases and expires pending runs, mayhem recover repairs a run's orphans on demand.

---

Exit Codes

Stable contract for scripts and CI (definition: src/mayhem/cli/exit_codes.py):

Code	Meaning
0	Success
1	General failure
2	Usage error (bad flags/arguments)
3	Configuration layering/validation failed
4	Spec/target validation failed
5	A safety gate refused the operation
6	Experiment ran and did not complete
7	Recovery/janitor left dirty state behind
8	Agent transport/runtime failure
9	External tool invocation failed structurally
10	Command prefix matched multiple commands

---

Architecture

The pipeline is staged so everything expensive is done up front and execution is as small as possible:

1. Discover — the topology provider builds a graph (services, hosts, dependency edges) from the compose blueprint and live containers.
2. Prepare — mayhem config layering (defaults → mayhem.yaml → profile → env → flags) plus topology, drift detection, and target revalidation.
3. Compile & plan — the drill spec becomes a frozen ExecutionPlan with step sequences, per-fault compensations, success criteria, and observability sources; every fault, target kind, capability, and duration is validated against the catalog.
4. Execute — the engine runs rounds (inject → observe → compensate) through the runtime adapter, evaluates criteria, collects observability, and writes step/event/lease rows with the governing-decision trace.
5. Recover & report — the janitor sweeps orphaned leases; status, history, and run summaries replay the evidence.

Documentation

Document	Contents
README.md	This file.
docs/drill-spec.md	The drill DSL reference — config, containers, execution, checks, success criteria, observability, and the full fault catalog.
docs/compensation.md	Fault compensation lifecycle — inject / undo / verify contracts, executor routing, marker conventions, and the per-fault template table.

Status

Area	Status
Domain models, configuration system	Complete
Topology discovery (Docker/Podman) + compose project filtering + drift detection	Complete
Fault catalog + registry + capability probing (toolkit)	Complete
Drill spec DSL (config / containers / execution / checks)	Complete
Deterministic + random planners, frozen plans	Complete
Success criteria + machine verdict	Complete
Declarative observability sources	Complete
Run engine with compensation + leases + janitor + recover	Complete
Safety gates + impact gate	Complete
CLI with prefix abbreviation, stable exit codes	Complete
SQLite persistence + restart, migrations (schema freeze)	Complete
Campaigns (multi-spec runs)	Complete
Tests (859 collected: 752 unit + 95 e2e + 12 integration), ruff, mypy (per-file strict)	Complete
Kubernetes execution	Planned (interface-only; see examples/k8s)
Web UI / REST API	Planned

---

Development

# Clone, then sync the dev dependency group (requires https://docs.astral.sh/uv/):
uv sync --group dev

# Run all tests
uv run pytest

# Lint
uv run ruff check src/

# Type check (strict is configured per module)
uv run mypy