Mayhem

Chaos engineering for Docker and Podman, with separately scoped Kubernetes planning and execution seams. The documented quickstart uses a compose blueprint. Mayhem compiles a declarative kind: drill document into a frozen, safety-gated plan, injects supported faults through a capability-aware toolkit, and derives the run verdict from recorded observations.

Everything lands in SQLite — steps, probes, criteria evaluations, decisions — so nothing is ever "trust me, it worked."

Kubernetes source support is layered and must not be collapsed into one "supported" claim: manifest planning, planner support, executor support, and live resolution are separate states. The legacy KubernetesAdapter remains unavailable, and catalog-only faults are not executable merely because they are defined. See the Kubernetes status section and the documentation authority index.

compose blueprint ─▶ topology graph ─▶ compile drill spec ─▶ frozen plan
                                                              │
                        verdict ◀── machine criteria ◀── inject → observe → recover
                                                              │
                                                SQLite evidence + decision trace

• Quickstart
• A Run in One Screen
• Authoring a Drill
• Configuration
• CLI Reference
• Campaigns
• What's new in v0.9.0
• Safety Model
• Exit Codes
• Architecture
• Status
• Development

Quickstart

Prerequisites

• Python 3.12+
• Docker with Compose v2, or Podman selected with --podman

The Kubernetes client ships as a default dependency of the distribution, so there is no separate install extra. The Kubernetes paths still need a reachable cluster, a valid kubeconfig context, and the capabilities a fault family declares. The checked-in documentation does not certify any live cluster.

Install

pip install mayhem-cli        # one bundle; console command is `mayhem`

Run the bundled example

A complete, self-contained six-service stack lives in examples/testCase/.

cd examples/testCase
docker compose up -d                          # 0. bring the stack up
mayhem discover topology --compose docker-compose.yml   # 1. blueprint → live graph
mayhem prepare validate mayhem.yaml --compose docker-compose.yml  # 2. compile + safety gates (injects nothing)
mayhem run mayhem.yaml --compose docker-compose.yml --execute  # 3. inject → observe → recover → verdict

Omit --compose and Mayhem auto-detects docker-compose.yml (or compose.yml) in the current directory.

validate and plan compile from the compose blueprint and do not inject a fault. Topology construction may still inspect an available container runtime for current-state data. run applies its impact gate and records the actual execution outcome.

The bundled spec exercises the compose-supported fault families used by the example stack. The examples/k8s directory is a separate manifest-backed planning example. Its presence does not prove live-cluster execution, and the current catalog is larger than the original nine-fault example. See Targets (cross-runtime) for the authored target syntax.

A Run in One Screen

A clean run needs no interpretation — the verdict is one line away.

$ mayhem run mayhem.yaml --compose docker-compose.yml --execute

# Run r-process-drill-8f2a1c
**status**: completed
**verdict**: pass
**success criteria**: ALL PASS
- [PASS] status:api-up.status PASS (expected 200, got 200)
- [PASS] latency:api-up.latency_ms PASS (31.2 <= 500.0)
**observations**: 2/2 sources collected
**decisions**: 5 governing decision revisions (snapshot in the run row)
**wall**: 32.4s

run r-process-drill-8f2a1c — inspect with `mayhem inspect history r-process-drill-8f2a1c`

Reading the transcript, top to bottom:

1. Status — the run's machine state: completed (or failed/aborted with a non-zero exit and dirty-lease warnings).
2. Verdict — pass/fail derived from the success criteria you declared in the spec; undecided when a run aborts or the spec has no success block.
3. Success criteria — every criterion evaluated against real observations, one line each, so a failure tells you exactly what drifted.
4. Observations — how many configured evidence sources actually delivered data (probes, logs, metrics …).
5. Decisions — the governing decision revisions that shaped this run; the decision trace is queryable afterward via mayhem inspect history.
6. Copy-paste handle — the run id for the follow-up commands below.

From there: mayhem inspect runs lists recent runs, mayhem inspect runs --run <run-id> shows full recorded metadata, and mayhem inspect history <run-id> replays the complete event journal (every step, probe sample, and lease for that run). Add --debug to mayhem run to stream each step live as it happens ([ok] injected proc.pause 10s into testcase-api, [ok] recovered ... (compensation ok)).

Authoring a Drill

A drill is one kind: drill YAML file — the complete DSL reference lives in docs/drill-spec.md. The shape:

apiVersion: "mayhem/v1"
kind: drill
name: checkout-recovery
hypothesis: "checkout stays available while cart writes are throttled"

config:
  risk_ceiling: high
  max_faults: 1
  timeout: 30m
  recovery: true

containers:
  cart-api:
    faults:
      - fault: net.latency
        duration: 10s
        params:
          delay_ms: 300
          jitter_ms: 25

execution:
  - sequential: [cart-api]
  - check:
      - http: http://cart-api:8080/_health
        expect:
          status: 200

Validate with mayhem prepare validate mayhem.yaml; unknown parameters, out-of-range durations, untargetable node kinds, and capability gaps are all compile-time errors — before anything is injected.

Faults are drawn from the catalog (net.latency, proc.kill, net.packet_loss, TLS failure, container pause, HTTP error injection, dependency and database faults, k8s.pod_kill, k8s.node_drain, …). The full per-fault reference — every id, its capabilities, risk level, and compensation contract — is in docs/drill-spec.md.

The containers: block above is the docker-family authoring shape. To fault a Kubernetes workload or node — or mix runtimes in one spec — use the cross-runtime targets: block instead (exactly one of containers: / targets: defines a spec); see Targets (cross-runtime).

Configuration

Runtime policies live in mayhem.yaml — the configuration file, distinct from a kind: drill spec — auto-detected in the cwd or given with --config. The effective view is one command away: mayhem prepare config show prints the resolved configuration and provenance; mayhem prepare config validate refuses unknown keys, a missing or wrong apiVersion, and out-of-range sections before anything runs.

apiVersion: mayhem/v1        # required; anything else is rejected
policy:
  allow_faults: null         # null = whole catalog; set to restrict
  deny_faults: []            # fault ids never injectable
  risk_ceiling: null         # tightened by the drill ceiling at plan time
  allow_critical: false      # config-side half of the critical opt-in
  critical_fault_acks: []    # per-fault acks; critical faults need allow_critical + ack + --allow-critical
blast_radius:
  max_services_pct: 50.0
  max_hosts: 2
  max_concurrent_faults: 3
  max_duration_per_fault_s: 300.0
  forbidden_fault_pairs: []  # pairs such as [net.packet_loss, net.bandwidth]
storage:
  path: mayhem.db
  artifacts_dir: .mayhem/artifacts
toolkit:
  binaries: {}               # pin a named tool's binary
runtime: docker              # docker | podman | kubernetes
target:
  containers: []             # explicit discovery targets without compose
kubernetes:
  context: null              # kubeconfig context
  namespace: null            # null = no namespace filter
recovery_grace: 300.0
log_level: INFO              # DEBUG | INFO | WARNING | ERROR
maniac:
  level: 2
  run_level: 10
  seed: null

Layering, in increasing precedence: built-in defaults → selected YAML → mayhem.{profile}.yaml → allowlisted environment variables → programmatic CLI overrides. Profile overlays are separate files selected with --profile NAME; there is no profiles: key inside the base file. The environment layer only honours MAYHEM_STORAGE_PATH, MAYHEM_ARTIFACTS_DIR, and MAYHEM_LOG_LEVEL. The current CLI uses --config and --profile to select layers; it does not expose a generic flag that maps arbitrary fields into configuration.

Drill-level config.risk_ceiling composes with the policy ceiling and can only tighten it.

The full configuration reference is in docs/config.md.

CLI surface

The active CLI is workflow-oriented: discover, prepare, experiment, run, inspect, recover, and extend. Guided init and doctor are active, and all legacy root commands and aliases have been removed. Exit codes, machine-readable fields, and database migrations remain stable.

See docs/product/cli-product-direction.md, docs/product/command-architecture.md, and docs/new-plan/README.md.

Root options precede the command. Unique prefixes work at the root and in the workflow groups.

Use each command's current --help output for accepted arguments. The full sub-command inventory, root options, and stable exit codes are in docs/reference/cli.md; that inventory is compared against src/mayhem/cli/command_registry.py by tests/unit/test_release_contract.py, so a command that is not documented here fails the suite.

Campaigns

A campaign groups authored drill-spec paths and runs them sequentially.

mayhem campaign create black-friday \
  --description "BFCM chaos" --hypothesis "checkout survives every single-fault failure"
mayhem campaign add-experiment black-friday mayhem.yaml
mayhem campaign add-experiment black-friday checkout-recovery.yaml
mayhem campaign start black-friday
mayhem campaign run black-friday --compose docker-compose.yml --execute

The current CLI creates campaigns in draft, approve moves a draft to approved, start begins a draft or approved campaign, run executes the stored spec paths, pause and resume stop and continue a running campaign, and archive or abort sets the corresponding terminal status. The current command surface does not expose campaign scheduling, priority ordering, or policy/window editing. See the command inventory.

What's new in v0.9.0

v0.9.0 is a truth-and-evidence release. It adds no new fault-injection power; it makes what Mayhem already does verifiable. Every new surface below is plan-only until explicitly executed, and every claim is recorded in evidence.

Two honesty rules are worth stating plainly, because they change what output means:

• No backend means no effect. start_load, stop_load, and notify have no executor yet. They report acknowledged_no_backend and the run is not clean — they never look like success.
• No live verification is claimed without a live run. The capability dashboard reports live=false for every row in this repository; live=true can only come from a recorded live run.

mayhem --version                     # installed version, or 0.0.0+source in a checkout
mayhem discover capabilities --explain k8s.pod_kill
mayhem inspect replay export RUN_ID --out bundle/
mayhem bundle verify bundle/

v0.9.0 documentation

Safety Model

• Risk ceilings. Every catalog fault carries a risk level; injection is refused when either the policy or the drill ceiling is exceeded. critical-risk faults (e.g. k8s.node_drain) need a triple opt-in: policy.allow_critical: true, a per-fault ack in policy.critical_fault_acks, and the --allow-critical CLI flag.
• Concurrency budget. max_faults caps simultaneously-injected faults; a wider parallel: step queues into rounds.
• Duration caps. Per-fault duration beyond the catalog maximum is a compile error.
• Capability gating. Faults declare the capabilities they need (docker engine, kubernetes_engine, net_admin, process control, …); the plan is proven against the live graph by the impact gate before run — never assumed.
• Compensation contracts. Reversible faults run their declared inverse; irreversible ones are followed by workload reconciliation. A failed round aborts-and-recovers its own faults first, then propagates.
• Lease hygiene. Fault rounds hold leases with a TTL; mayhem janitor sweeps orphaned leases and expires pending runs, mayhem recover repairs a run's orphans on demand.

Exit Codes

The stable identifiers and numeric values are defined in src/mayhem/cli/exit_codes.py and documented in docs/reference/cli.md. The deterministic documentation test rejects identifiers that are not declared in source.

Architecture

The pipeline is staged so everything expensive is done up front and execution is as small as possible:

1. Discover — compose discovery builds a Docker/Podman graph. Kubernetes has separate live-discovery and offline-manifest providers; the manifest provider creates logical placeholders, not live pod selections.
2. Prepare — mayhem prepare config layering (defaults → selected YAML → separate profile overlay → allowlisted environment values → programmatic overrides), plus topology, drift detection, and target revalidation.
3. Compile & plan — the drill spec becomes a frozen ExecutionPlan with step sequences, per-fault compensations, success criteria, and observability sources; fault, target, capability, and duration inputs are validated against the current models.
4. Execute — supported runtimes execute inject → hold → compensate rounds and record evidence. Kubernetes planner/executor presence does not by itself establish a reachable cluster or an available capability.
5. Recover & report — mayhem janitor sweeps orphaned leases; mayhem recover status|plan|execute, mayhem inspect runs, and mayhem inspect history replay recorded evidence.

Documentation

Status

Kubernetes status

The examples/k8s README and documentation authority index explain the separate Kubernetes states. Historical discovery reports and the dated grounding log are retained for traceability and must not be treated as live-cluster evidence.

Development

# Clone, then sync the dev dependency group (requires https://docs.astral.sh/uv/):
uv sync --group dev

# Run all tests
uv run pytest

# Check the release truth baseline (docs, Justfile, packaging vs. source)
uv run pytest tests/unit/test_release_contract.py

# Build the sdist and wheel
uv build

# Lint
uv run ruff check src/

# Type check (strict is configured per module)
uv run mypy