microburst

site · pypi · releases

AWS failure injection that your SDK actually believes.
Throttling, latency, timeouts and resets — in the exact wire format AWS uses,
so retry, backoff and circuit-breaker code gets exercised for real.

The problem

Your app handles ThrottlingException and retries with backoff — or so you hope. The only way to know is to make AWS actually throttle you, and until now your options were:

microburst is the missing piece: a standalone, protocol-aware proxy that works against any AWS-compatible endpoint — MiniStack, moto, or real AWS — and returns failures the SDK can't tell apart from the real thing.

Why "protocol-aware" matters

SDKs decide whether to retry by parsing the error code out of the response body, and whether that code is retryable depends on the status too. S3 SlowDown at HTTP 400 is a terminal client error; at 503 the SDK backs off and retries. Get the shape wrong and you're testing a failure AWS never produces.

microburst reads the botocore service models — the same definitions the SDK uses — so injected errors carry the right code, the right XML/JSON envelope, and the right status.

Install

pip install microburst        # or: uvx microburst
pip install microburst[tui]   # + TUI dashboard (rich)
pip install microburst[h2]    # + HTTP/2 upstream transport (httpx)
pip install microburst[otel]  # + OpenTelemetry fault spans

Docker:

docker run --rm -p 9999:9999 ghcr.io/pingedbrain/microburst:latest \
  --upstream http://host.docker.internal:4566

Docker Compose (microburst + MiniStack wired together):

docker compose -f examples/docker-compose.yml up

Runnable failure scenarios — throttled writers, timeout vs retry-budget, poison queues, stream cuts, mid-upload resets, generic HTTP deps — live in examples/.

GitHub Action

Drop fault injection into any workflow — microburst runs as a step container and exports AWS_ENDPOINT_URL for you:

jobs:
  chaos-tests:
    runs-on: ubuntu-latest
    services:
      ministack:
        image: ministackorg/ministack:latest
        ports: ["4566:4566"]
    steps:
      - uses: actions/checkout@v4
      - uses: pingedbrain/microburst@v0.3.0
        with:
          upstream: http://localhost:4566
          config: .github/chaos.yml   # optional rules file
      - run: pytest                 # AWS_ENDPOINT_URL already set

Quickstart

microburst --upstream http://localhost:4566   # your emulator, e.g. MiniStack

export AWS_ENDPOINT_URL=http://localhost:9999
python your_app.py        # all AWS calls now flow through microburst

Inject throttling at runtime:

curl -X PATCH localhost:9999/_microburst/rules -d '[
  {"service": "dynamodb", "probability": 0.3,
   "error": {"code": "ProvisionedThroughputExceededException"}}
]'

Or fire a preset:

curl -X POST localhost:9999/_microburst/presets/ddb-throttle

Then watch exactly what fired — chaos you can audit:

curl localhost:9999/_microburst/fired
# → [{"rule": "...", "service": "dynamodb", "operation": "PutItem",
#     "action": "error:ProvisionedThroughputExceededException", ...}]

Or watch it live in the terminal:

microburst dashboard            # TUI: rules + live fault stream (needs [tui])

Rules

- service: dynamodb          # SigV4 credential-scope name, "*" for all
  operation: PutItem         # optional; resolved per AWS protocol
  region: us-east-1          # optional
  resource: orders           # substring of table/bucket/queue/…
  headers:                   # optional; all must match (substring)
    x-amz-acl: public-read   # e.g. only canned-ACL puts
    x-amz-copy-source: ""    # "" = presence check (e.g. CopyObject)
  body: "TableName == 'orders'"        # jmespath on JSON/form body — truthy = match
  rate: {count: 5, window_s: 60}       # at most N fires per rolling window
  sequence: {fail: 3, pass: 2}         # fail 3, pass 2, repeat
  probability: 0.5           # default 1.0
  deterministic: true        # hash the request identity — the same resource
                             # always lands on the same side of p (reproducible
                             # "this bucket always fails" without RNG seeds)
  times: 3                   # fire at most N times, then pass through
  ttl_s: 120                 # rule expires N seconds after creation
  active_at: "2026-01-01T00:00:00Z"  # start matching at this time
  until: 1893456000          # stop matching then (epoch or ISO-8601)
  error:
    code: SlowDown           # omit → samples a plausible modeled exception
    status: 503              # omit → modeled/curated AWS status
    message: "slow down"
    fields:                  # extra error-shape members, rendered per
      BucketName: my-bucket  # protocol (json members / XML elements)
  latency: {min: 500, max: 2000}   # ms; or a bare number, or a distribution:
                                   # {dist: gaussian, mean: 500, stddev: 100,
                                   #  min: 100, max: 2000}
                                   # {dist: spike, min: 50, max: 100,
                                   #  spike_ms: 5000, spike_p: 0.05}
                                   # {preset: dynamodb} — measured per-service
                                   #  baseline (see below); explicit keys in the
                                   #  same block override preset values
  timeout_ms: 30000                # hold the connection, then 504
  reset: true                      # abort the TCP connection
  response:                        # post-forward: mutate the upstream response
    truncate_frac: 0.5             # valid envelope, body cut short (or
                                   # truncate_bytes: N)
    abort_frac: 0.3                # send 30%, then kill the connection
                                   # mid-stream (or abort_bytes: N)
    corrupt_bytes: 16              # flip N bytes — 200 OK, wrong payload
    bandwidth_kbps: 64             # cap downstream throughput
    event_error:                   # eventstream (Kinesis SubscribeToShard,
      code: ThrottlingException    #  S3 Select): splice a well-formed
      message: "slowed mid-stream" #  :error frame mid-stream — terminal
      after_frames: 3              #  for the stream, after N real frames
    event_frames:                  # frame-level eventstream surgery:
      - at: 1                      #  `at` counts upstream frames (0-based)
        drop: true                 #  drop that frame
      - at: 3
        inject:                    #  emit a custom frame before index 3
          event_type: Stats
          payload: '{"BytesScanned": 42}'   # str, dict, or payload_b64
      - at: 4
        payload: '{"rew": 1}'      #  replace payload, CRCs recomputed
      - at: 5
        bad_crc: true              #  broken CRC → SDK checksum error
      - at: 6
        cut: 0.5                   #  emit half the frame, then EOF
      - at: 8
        error: {code: ThrottlingException}  # terminal :error frame
    set_headers:                   # mutate response headers — wrong CT on
      Content-Type: text/plain     # a 200, added x-amz-*, etc.
    strip_headers: [ETag]          # drop response headers entirely
  request:                         # pre-forward: fault the client→proxy upload
    slow_upload: {rate_kbps: 8}    # read the client body at ≤8 KiB/s — stalls
                                   # the SDK's write path (exercises write/
                                   # socket timeouts); the body still
                                   # arrives whole upstream
    cut_upload: {after_bytes: 1024}  # after N bytes of the upload are
                                   # consumed, reset the client connection —
                                   # ECONNRESET mid-PUT, nothing forwarded
                                   # (or after_frac: 0.5, or
                                   #  after_messages: N — framed bodies only:
                                   #  count messages and reset on a frame
                                   #  boundary, eventstream / grpc*)
    corrupt_upload: {at_message: 3}  # framed bodies: forward N-1 messages
                                   # verbatim, poison message N's
                                   # checksum/length so the UPSTREAM's
                                   # parser rejects it — the client stays
                                   # connected for the upstream's error

times: 1 is the sleeper feature — "fail exactly once, then let the retry succeed" verifies your retry path end-to-end instead of just proving errors surface.

latency: {preset: <service>} picks a measured per-service baseline (dynamodb, s3, sqs, sns, lambda, kinesis, iam, ec2, cloudformation, ssm, secretsmanager, sts, logs, firehose, events, stepfunctions, kms, athena, route53, cloudfront, glacier, wafv2, elbv2, apigateway, pinpoint). The numbers are a real AWS observation — read-only ops probed against live us-east-1 in 2026-02, n=8 per service — encoded as each service's service-side residual: observed p50 minus the ~155ms host→region network floor, because your own network isn't microburst's to simulate. The gaussian stddev (max(5, 0.35·mean)) is a modeling inference, not a measurement — service-side variance isn't recoverable through network noise. Treat presets as order-of-magnitude baselines, not SLAs; refresh them with tools/latency_probe.py (real AWS only).

Fault ordering: a fired rule applies request: faults first — the upload happens before any response can exist, so a cut_upload link dies before an error: envelope could be sent — then latency → reset → timeout → error, then the forward, then response: mutations on the way back.

One fidelity note on request: faults: microburst buffers request bodies when detection needs them (non-streaming ops ≤4 MiB, body: matchers, cassette mode). A buffered upload has already finished on the client's side, so slow_upload can't stall its write — the pacing moves to the upstream send — and cut_upload lands on the client's read path (same observable as reset: true). Streaming uploads (S3 PutObject, UploadPart, payloads >4 MiB) get the real thing: write-path backpressure and a true mid-upload reset.

after_messages / corrupt_upload need a framed Content-Type: application/vnd.amazon.eventstream (prelude-parsed frames) or application/grpc* (5-byte length prefix). On any other type they're no-ops — the request forwards and the fired event's note field says so (content-type not a framed stream). Frame parsing is defensive: a malformed prelude/prefix stops message counting, falls back to the byte thresholds (which still apply if configured), and forwards verbatim on corrupt_upload; every degraded outcome lands in note with frames_seen detail.

Presets

ddb-throttle · flaky-s3 · slow-lambda · kms-outage · sqs-backlog · regional-failover · network-jitter · gateway-storm · expired-token · clock-skew · bad-signature

Control API

Load rules at startup with microburst --config chaos.yml (see examples/chaos.yml); add --watch to hot-reload the file on every save — the file replaces the whole ruleset each reload.

PostgreSQL wire mode

microburst also speaks the PostgreSQL v3 wire protocol — a TCP proxy that injects SQLSTATE-correct ErrorResponses and connection faults between your app and any Postgres server. PG is a sibling transport, not a second use of the HTTP pipeline: the same rules engine, fired log, control API, and stats serve both modes.

microburst --protocol postgres --port 15432 --upstream localhost:5432
# control API is its own HTTP listener (default :9999, --control-port)

psql "host=localhost port=15432 dbname=mydb" -c "select 1"
# → flows through the proxy to localhost:5432

Rule matching for pg: service: postgres, operation: is the lowercased SQL verb (select, insert, begin, …) or startup for the connect handshake; resource: matches a table-ish token; sql: is a case-insensitive regex against the raw query text.

# refuse connections like a maxed-out pool
- service: postgres
  operation: startup
  error: {sqlstate: "53300", severity: FATAL, message: "too many connections"}

# serialization failure — psql shows: ERROR:  could not serialize access
- service: postgres
  operation: select
  sql: "into orders"
  error: {sqlstate: "40001", message: "could not serialize access"}

# slow, then canceled — a realistic statement_timeout
- operation: update
  latency: {min: 800, max: 1500}
  error: {sqlstate: "57014", message: "canceling statement due to statement timeout"}

# server dies mid-ResultSet: 2 real rows, then the TCP link aborts
- operation: select
  partial_rows: 2

- operation: select
  timeout: true     # hang until the client gives up
  # or: reset: true  # TCP RST

Severity is the fidelity. severity: FATAL (or PANIC) sends the ErrorResponse then closes the connection — exactly what real PostgreSQL does — and is the only error class safe to inject inside an open transaction. A plain ERROR is injected only while the session is idle (upstream's last ReadyForQuery said I): inside a transaction (T/E) the rule is skipped and the query forwards untouched, because an error the upstream never saw would desync client and server tx state. Skips are recorded on the fired event's note (skipped: in-transaction).

Startup errors always close the session after the ErrorResponse — a refused connection has no session to return to. timeout_ms: N stalls a query for N ms; timeout: true stalls until the client disconnects. Extended protocol (Parse…Sync batches, including named prepared statements learned per connection) is decided at batch granularity. Every fired fault lands in /_microburst/fired and /metrics with service="postgres", operation = verb, and the first ~80 chars of SQL in path.

Limitations (honest list):

• Auth is passthrough only — SCRAM works because credentials are relayed uninspected; the proxy never synthesizes AuthenticationOk.
• SSLRequest/GSSENCRequest are refused with N (plaintext only).
• COPY streams relay verbatim but are never intercepted.
• No replication protocol (walsender/CopyBothResponse), no FunctionCall interception (it forwards + relays), no per-statement error replay like 25P02 emulation inside a transaction — injecting into an open tx needs upstream tx tracking beyond the status byte.
• Only the first statement of a multi-statement Query is classified.

Real AWS upstreams

microburst --upstream https://dynamodb.us-east-1.amazonaws.com

Requests are re-signed with your credentials automatically for amazonaws.com upstreams (--no-resign to disable). Useful for game days against staging accounts. Add --http2 (needs microburst[h2]) to talk HTTP/2 to the upstream — AWS endpoints negotiate it via ALPN.

Cassettes: record & replay

Record real upstream traffic once, replay it forever — with rules still injecting faults on top:

microburst --upstream https://dynamodb.us-east-1.amazonaws.com --record cass/
microburst --replay cass/ --config chaos.yml   # no upstream contact

Entries are keyed by sha256(method + path + query + body) — headers are excluded so signatures/timestamps don't matter. Replays are byte-exact (status, headers, body); response faults and injected errors apply normally, so a replayed stream is deterministic underneath and chaotic on top.

Fidelity vs real AWS

Error envelopes aren't guessed — they're diffed against live AWS captures. tools/live_fidelity.py records raw wire responses from real AWS (read-only probes against nonexistent resources, credentials from your profile/env) and diffs them against what render_error produces:

AWS_PROFILE=you microburst fidelity capture   # raw wire captures
microburst fidelity capture --region eu-west-1 --dir eu/  # any region
microburst fidelity report                    # → fidelity/REPORT.md
microburst fidelity snapshot                  # model digests (no creds)

# emulator conformance — same probes, diffed against the AWS goldens
microburst fidelity capture --endpoint-url http://localhost:4566 --dir ms/
microburst fidelity conform --emu ms/ --aws fidelity/   # → CONFORM.md

# compare any two capture sets directly
microburst fidelity diff ms/ other-emulator/            # → DIFF.md

The evidence stays fresh without anyone owning AWS credentials: a weekly model-drift workflow regenerates fidelity/models_snapshot.json against the latest botocore (AWS's models are upstream of the wire) and opens an issue when error shapes, routes, or protocol metadata move — that's the signal to re-capture. fidelity/protocol/ vendors AWS's own protocol-compliance fixtures (from botocore's conformance suite), so the envelopes are also checked against AWS-authored wire expectations on every test run.

A second weekly workflow, emulator-conformance, re-runs the same probe set against a LocalStack service container and conforms it against the committed goldens — an informational measurement, never a merge gate. The score table lands in the run's step summary; CONFORM.md plus the raw captures upload as the localstack-conformance run artifact.

The committed report (fidelity/REPORT.md) shows 28/28 probes matching AWS on status, parsed Error.Code, Content-Type, and envelope shape (XML element paths, __type namespacing) — including the details that matter to SDK retry behavior: x-amz-json-1.1 content types, com.amazonaws.*-namespaced __type, rest-json x-amzn-ErrorType headers, SQS's AWS.SimpleQueueService.* query-compat namespace, Route53's text/xml, empty-body HEAD errors, and Athena's AthenaErrorCode/ErrorCode semantic fields.

The envelopes are also region-invariant: the same probe set captured in every enabled region of a real account (17 regions) conforms 28/28 — the only per-region difference observed is service availability (e.g. Pinpoint has no endpoint in 5 regions), never the wire shape.

capture also runs SUCCESS_PROBES — read-only list/describe calls that return 200 on a fresh account — recording "kind": "success" captures across all six wire families (success ops that collide with an error probe name land in __ok-suffixed files). report skips them with a note (microburst forwards success bodies verbatim — there is nothing to render), while conform/diff compare them on status + Content-Type + envelope shape, which is where emulator success-shape conformance pays off. The 25 committed success goldens are real AWS captures with content_scrubbed — leaf values replaced by type-shaped placeholders (resource names, ids, dates) while the element structure stays verbatim. Caveat: success shapes are data-dependent — an empty Buckets list vs a populated one is a real path-set diff, so emulator conformance on success bodies reflects both format and cardinality.

Multi-SDK matrix

tools/sdk-matrix/ runs real SDK clients — boto3, aws-sdk-js-v3, aws-sdk-go-v2, aws-sdk-java-v2 — against a live microburst and verifies what each SDK parsed, not what we sent: error code, HTTP status, and retry attempts (measured twice — client-side and via the proxy's fired log). All 16 scenario cells pass in CI on every push; the README in that directory documents the one genuine cross-SDK divergence (codeless HEAD errors: boto3 reports "503", Go "ServiceUnavailable", JS "Unknown", Java null — the same labels they produce against real AWS).

Caveats

• Downstream is HTTP/1.1 (AWS SDKs don't speak h2 to the client anyway); upstream can be HTTP/2 with --http2. Event-stream APIs support mid-stream frame surgery via response.event_frames (drop, repayload, corrupt, inject, cut, :error — all with valid framing/CRCs unless bad_crc is the point).
• Bodies > 4 MiB are streamed uninspected (resource matchers won't apply; service/operation still do for REST services).
• The control API is unauthenticated — bind it to localhost only.
• SigV4A (multi-region) requests parse fine; rules see region: "*".

Demo

With MiniStack (or any emulator) on :4566:

python demo.py   # orders pipeline → microburst → MiniStack, scripted fault windows

You'll see DynamoDB puts retry through injected throttling, SQS publishes degrade under latency, S3 reads ride out SlowDown, and the pipeline recover when faults clear — plus the fired-fault ledger at the end. A real run (endpoints configurable via MINISTACK_URL/MICROBURST_URL):

⚡ FAULT INJECTED → dynamodb PutItem throttles 35%
    2.1s  order-6    put     56ms ↻ retried ×1 (throttled)  publish   2ms  s3 ✓
    6.8s  order-15   put    757ms ↻ retried ×4 (throttled)  publish   2ms  s3 ✓

⚡ FAULT INJECTED → S3 GetObject SlowDown 50%
   17.7s  order-21   put    356ms ↻ retried ×3   publish  2111ms (slow)  s3 ↻ ×1

⚡ FAULTS CLEARED → recovery
   20.4s  order-23   put      2ms  publish      2ms   s3 ✓     2ms

━━━ fired log (what microburst actually did) ━━━
   17× dynamodb:PutItem → error:ProvisionedThroughputExceededException
    3× s3:GetObject → error:SlowDown
    5× sqs:SendMessage → latency:1009–2192ms

━━━ outcome ━━━
  orders processed:      31
  ddb throttled+retried: 11 (SDK absorbed — app never saw an error)
  ddb hard failures:     0

Contributing & community

• Contributing · Code of Conduct · Security
• This repo adopts Apache Magpie for agent-assisted maintainership (see .apache-magpie.lock).

License

MIT — go break your own stuff before production does.