microburst

site · pypi · releases

AWS failure injection that your SDK actually believes.
Throttling, latency, timeouts and resets — in the exact wire format AWS uses,
so retry, backoff and circuit-breaker code gets exercised for real.

The problem

Your app handles ThrottlingException and retries with backoff — or so you hope. The only way to know is to make AWS actually throttle you, and until now your options were:

microburst is the missing piece: a standalone, protocol-aware proxy that works against any AWS-compatible endpoint — MiniStack, moto, or real AWS — and returns failures the SDK can't tell apart from the real thing.

Why "protocol-aware" matters

SDKs decide whether to retry by parsing the error code out of the response body, and whether that code is retryable depends on the status too. S3 SlowDown at HTTP 400 is a terminal client error; at 503 the SDK backs off and retries. Get the shape wrong and you're testing a failure AWS never produces.

microburst reads the botocore service models — the same definitions the SDK uses — so injected errors carry the right code, the right XML/JSON envelope, and the right status.

Install

pip install microburst        # or: uvx microburst
pip install microburst[tui]   # + TUI dashboard (rich)
pip install microburst[h2]    # + HTTP/2 upstream transport (httpx)
pip install microburst[otel]  # + OpenTelemetry fault spans

Docker:

docker run --rm -p 9999:9999 ghcr.io/pingedbrain/microburst:latest \
  --upstream http://host.docker.internal:4566

Docker Compose (microburst + MiniStack wired together):

docker compose -f examples/docker-compose.yml up

GitHub Action

Drop fault injection into any workflow — microburst runs as a step container and exports AWS_ENDPOINT_URL for you:

jobs:
  chaos-tests:
    runs-on: ubuntu-latest
    services:
      ministack:
        image: ministackorg/ministack:latest
        ports: ["4566:4566"]
    steps:
      - uses: actions/checkout@v4
      - uses: pingedbrain/microburst@v0.3.0
        with:
          upstream: http://localhost:4566
          config: .github/chaos.yml   # optional rules file
      - run: pytest                 # AWS_ENDPOINT_URL already set

Quickstart

microburst --upstream http://localhost:4566   # your emulator, e.g. MiniStack

export AWS_ENDPOINT_URL=http://localhost:9999
python your_app.py        # all AWS calls now flow through microburst

Inject throttling at runtime:

curl -X PATCH localhost:9999/_microburst/rules -d '[
  {"service": "dynamodb", "probability": 0.3,
   "error": {"code": "ProvisionedThroughputExceededException"}}
]'

Or fire a preset:

curl -X POST localhost:9999/_microburst/presets/ddb-throttle

Then watch exactly what fired — chaos you can audit:

curl localhost:9999/_microburst/fired
# → [{"rule": "...", "service": "dynamodb", "operation": "PutItem",
#     "action": "error:ProvisionedThroughputExceededException", ...}]

Or watch it live in the terminal:

microburst dashboard            # TUI: rules + live fault stream (needs [tui])

Rules

- service: dynamodb          # SigV4 credential-scope name, "*" for all
  operation: PutItem         # optional; resolved per AWS protocol
  region: us-east-1          # optional
  resource: orders           # substring of table/bucket/queue/…
  headers:                   # optional; all must match (substring)
    x-amz-acl: public-read   # e.g. only canned-ACL puts
    x-amz-copy-source: ""    # "" = presence check (e.g. CopyObject)
  body: "TableName == 'orders'"        # jmespath on JSON/form body — truthy = match
  rate: {count: 5, window_s: 60}       # at most N fires per rolling window
  sequence: {fail: 3, pass: 2}         # fail 3, pass 2, repeat
  probability: 0.5           # default 1.0
  deterministic: true        # hash the request identity — the same resource
                             # always lands on the same side of p (reproducible
                             # "this bucket always fails" without RNG seeds)
  times: 3                   # fire at most N times, then pass through
  ttl_s: 120                 # rule expires N seconds after creation
  error:
    code: SlowDown           # omit → samples a plausible modeled exception
    status: 503              # omit → modeled/curated AWS status
    message: "slow down"
  latency: {min: 500, max: 2000}   # ms; or a bare number, or a distribution:
                                   # {dist: gaussian, mean: 500, stddev: 100,
                                   #  min: 100, max: 2000}
                                   # {dist: spike, min: 50, max: 100,
                                   #  spike_ms: 5000, spike_p: 0.05}
  timeout_ms: 30000                # hold the connection, then 504
  reset: true                      # abort the TCP connection
  response:                        # post-forward: mutate the upstream response
    truncate_frac: 0.5             # valid envelope, body cut short (or
                                   # truncate_bytes: N)
    abort_frac: 0.3                # send 30%, then kill the connection
                                   # mid-stream (or abort_bytes: N)
    corrupt_bytes: 16              # flip N bytes — 200 OK, wrong payload
    bandwidth_kbps: 64             # cap downstream throughput
    event_error:                   # eventstream (Kinesis SubscribeToShard,
      code: ThrottlingException    #  S3 Select): splice a well-formed
      message: "slowed mid-stream" #  :error frame mid-stream — terminal
      after_frames: 3              #  for the stream, after N real frames

times: 1 is the sleeper feature — "fail exactly once, then let the retry succeed" verifies your retry path end-to-end instead of just proving errors surface.

Presets

ddb-throttle · flaky-s3 · slow-lambda · kms-outage · sqs-backlog · regional-failover · network-jitter · gateway-storm

Control API

Load rules at startup with microburst --config chaos.yml (see examples/chaos.yml).

Real AWS upstreams

microburst --upstream https://dynamodb.us-east-1.amazonaws.com

Requests are re-signed with your credentials automatically for amazonaws.com upstreams (--no-resign to disable). Useful for game days against staging accounts. Add --http2 (needs microburst[h2]) to talk HTTP/2 to the upstream — AWS endpoints negotiate it via ALPN.

Cassettes: record & replay

Record real upstream traffic once, replay it forever — with rules still injecting faults on top:

microburst --upstream https://dynamodb.us-east-1.amazonaws.com --record cass/
microburst --replay cass/ --config chaos.yml   # no upstream contact

Entries are keyed by sha256(method + path + query + body) — headers are excluded so signatures/timestamps don't matter. Replays are byte-exact (status, headers, body); response faults and injected errors apply normally, so a replayed stream is deterministic underneath and chaotic on top.

Fidelity vs real AWS

Error envelopes aren't guessed — they're diffed against live AWS captures. tools/live_fidelity.py records raw wire responses from real AWS (read-only probes against nonexistent resources, credentials from your profile/env) and diffs them against what render_error produces:

AWS_PROFILE=you microburst fidelity capture   # raw wire captures
microburst fidelity report                    # → fidelity/REPORT.md

The committed report (fidelity/REPORT.md) shows 28/28 probes matching AWS on status, parsed Error.Code, and Content-Type — including the details that matter to SDK retry behavior: x-amz-json-1.1 content types, com.amazonaws.*-namespaced __type, rest-json x-amzn-ErrorType headers, SQS's AWS.SimpleQueueService.* query-compat namespace, Route53's text/xml, and empty-body HEAD errors.

Multi-SDK matrix

tools/sdk-matrix/ runs real SDK clients — boto3, aws-sdk-js-v3, aws-sdk-go-v2, aws-sdk-java-v2 — against a live microburst and verifies what each SDK parsed, not what we sent: error code, HTTP status, and retry attempts (measured twice — client-side and via the proxy's fired log). All 16 scenario cells pass in CI on every push; the README in that directory documents the one genuine cross-SDK divergence (codeless HEAD errors: boto3 reports "503", Go "ServiceUnavailable", JS "Unknown", Java null — the same labels they produce against real AWS).

Caveats

• Downstream is HTTP/1.1 (AWS SDKs don't speak h2 to the client anyway); upstream can be HTTP/2 with --http2. Event-stream APIs support mid-stream :error frame injection via response.event_error — arbitrary per-payload mutation isn't implemented.
• Bodies > 4 MiB are streamed uninspected (resource matchers won't apply; service/operation still do for REST services).
• The control API is unauthenticated — bind it to localhost only.
• SigV4A (multi-region) requests parse fine; rules see region: "*".

Demo

With MiniStack (or any emulator) on :4566:

python demo.py   # orders pipeline → microburst → MiniStack, scripted fault windows

You'll see DynamoDB puts retry through injected throttling, SQS publishes degrade under latency, S3 reads ride out SlowDown, and the pipeline recover when faults clear — plus the fired-fault ledger at the end.

Contributing & community

• Contributing · Code of Conduct · Security
• This repo adopts Apache Magpie for agent-assisted maintainership (see .apache-magpie.lock).

License

MIT — go break your own stuff before production does.