erv2-itest 0.2.1

═════════════════════════════════════════════════════

   🧪  E R V 2 - I T E S T  🐳

   unattended integration tests for ERv2 modules

═════════════════════════════════════════════════════

✨ What is this?

erv2-itest is an unattended integration-test runner for ERv2 modules. Point it at a YAML scenario — or let it auto-discover every scenario in your repo — and it drives a sequence of docker run steps against a real module image (Apply, Apply again, Destroy), checking each step's exit code and log output against what you expect. No manual docker run + eyeballing logs, no guessing whether a reconcile loop actually did the right thing, and no credential files to hand-manage — it fetches AWS credentials from Vault on demand and falls back to sensible defaults (image name, config, output paths) so a module usually needs little to no setup to get its first scenario running.

🎬 Demo

uv run erv2-itest --scenarios-dir demo -k quick --no-dry-run --output-dir .erv2-itests-demo
────────────────────────────────── 🚀 demo-quick-win ──────────────────────────────────
🧪 Scenario: demo-quick-win
🆔 Run ID:   erv2it-20260917074749-0967
⚙️ Mode:     erv2
🎯 Target:   erv2it-demo:latest
🏗️ Built:    2026-09-17 07:18:47 UTC
📄 Log:      .erv2-itests-demo/logs/erv2it-20260917074749-0967.log
📊 Summary:  .erv2-itests-demo/logs/erv2it-20260917074749-0967.json

  ── 👣 Step: apply ──
    $ docker run --rm --name erv2it-20260917074749-0967-apply --mount type=bind,source=.erv2-itests-demo/runs/erv2it-20260917074749-0967/input.json,target=/inputs/input.json --mount type=bind,source=demo/credentials,target=/credentials --mount type=bind,source=.erv2-itests-demo/runs/erv2it-20260917074749-0967,target=/work -e DRY_RUN=False -e ACTION=Apply erv2it-demo:latest
    ACTION=Apply DRY_RUN=False
    loop count: 1
    still in progress
  ✅ apply

  ── 👣 Step: apply settles ──
    $ docker run --rm --name erv2it-20260917074749-0967-apply-settles --mount type=bind,source=.erv2-itests-demo/runs/erv2it-20260917074749-0967/input.json,target=/inputs/input.json --mount type=bind,source=demo/credentials,target=/credentials --mount type=bind,source=.erv2-itests-demo/runs/erv2it-20260917074749-0967,target=/work -e DRY_RUN=False -e ACTION=Apply erv2it-demo:latest
    ACTION=Apply DRY_RUN=False
    loop count: 2
    No changes. Your infrastructure matches the configuration.
  ✅ apply settles

  ── 👣 Step: destroy ──
    $ docker run --rm --name erv2it-20260917074749-0967-destroy --mount type=bind,source=.erv2-itests-demo/runs/erv2it-20260917074749-0967/input.json,target=/inputs/input.json --mount type=bind,source=demo/credentials,target=/credentials --mount type=bind,source=.erv2-itests-demo/runs/erv2it-20260917074749-0967,target=/work -e DRY_RUN=False -e ACTION=Destroy erv2it-demo:latest
    ACTION=Destroy DRY_RUN=False
    Destroy complete! Resources: 0 added, 0 changed, 1 destroyed.
  ✅ destroy

🎉 RESULT: PASS (demo-quick-win)
────────────────────────────────── 1 passed in 0.73s ──────────────────────────────────
            🏁 test results
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┓
┃ Scenario / Step ┃  Status   ┃ Reason ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━┩
│ demo-quick-win  │ ✅ PASSED │        │
│   apply         │ ✅ PASSED │        │
│   apply settles │ ✅ PASSED │        │
│   destroy       │ ✅ PASSED │        │
└─────────────────┴───────────┴────────┘

🌟 Features

• 📄 YAML scenarios — describe a sequence of steps, each with an expected exit code and optional log-content assertions.
• 🤖 Fully unattended — a scenario run's pass/fail is a single process exit code, ready for CI or an agent to check without parsing logs.
• 🔍 Test-suite style discovery — run erv2-itest with no arguments and it discovers and runs every scenario under integration-tests/, including subdirectories, with -k selection and -x/--fail-fast.
• 🔐 No credential files lying around — omit credentials_file and erv2-itest generates one from Vault on demand, cached locally, never committed.
• 🛡️ Interrupt-safe — Ctrl-C or a timeout kills the actual container, not just the local CLI client, so a real AWS operation never keeps running unnoticed.
• 🩹 Crash recovery — -k "::step-name" reruns just one step (e.g. cleanup), and --run-id targets the exact same resources a crashed run created.
• 🧩 Module-agnostic — knows only that a module's input is data/provision-shaped JSON and that it reports success/failure via exit code + stdout. Your scenario YAMLs and base-input templates live in your own module's repo, not here.
• 🔌 Pluggable execution backends — Docker/ERv2 today, with the architecture already in place for a native Terraform mode (no ERv2 wrapper) to land without changing how scenarios are written.
• 🐳 Per-module Docker customization — force docker or podman, or pass extra environment variables into the container, on a per-module basis.
• 📊 Structured, persisted results — every real run leaves a full transcript and a machine-readable JSON summary behind under .erv2-itests/logs/, independent of pass/fail.
• 🙈 Keeps your .gitignore honest — automatically adds its own output directory the first time you run for real, so generated artifacts (including cached credentials) never end up committed.

🚀 Quickstart

Add it as a dev dependency of your ERv2 module repo:

uv add --dev erv2-itest

Write a scenario (see SCENARIO_SCHEMA.md for the full format):

name: my-module-happy-path
module:
  image: my-erv2-module:latest
  # no credentials_file - generated from Vault on demand, see Configuration below
base_input: integration-tests/base_input.json
steps:
  - name: apply
    expect:
      exit_code: 0
      log_contains: ["Apply complete!"]
cleanup:
  name: destroy
  action: Destroy
  expect:
    exit_code: 0

Every field beyond name and steps has a sensible fallback, so even this runs for real by default:

name: my-module-happy-path
steps:
  - name: apply
    expect:
      exit_code: 0

erv2-itest fills in the rest: image <current directory name>:prod, an empty base input, and Vault-generated credentials (see "Configuration" below).

erv2-itest runs for real by default - --no-dry-run is implicit. Preview first if you want to check what it would do without touching anything:

uv run erv2-itest integration-tests/my-module-happy-path.yaml --dry-run

Then run it for real (no flag needed):

uv run erv2-itest integration-tests/my-module-happy-path.yaml

Or drop it under integration-tests/ (subdirectories are fine) and run every scenario in the repo at once, like a test suite:

uv run erv2-itest

Run artifacts (per-run Terraform working dir, full log, JSON summary, cached Vault credentials) land under .erv2-itests/ in your current directory - automatically added to .gitignore the first time you run for real.

📋 Scenario format

See SCENARIO_SCHEMA.md for the full schema reference.

⚙️ Configuration

Settings are resolved in this order - CLI flag > ERV2_ITEST_* env var > .erv2_itest.yml > built-in default:

# .erv2_itest.yml, in your repo root
output_dir: .erv2-itests
scenarios_dir: integration-tests
default_mode: erv2
log_level: INFO
target_account: app-sre/creds/terraform/ter-int-dev/config
tf_state_account: app-sre/creds/terraform/ter-int-dev/config
# default_module / default_terraform / default_base_input: let every scenario in this
# repo omit an identical module:/terraform:/base_input: block - see SCENARIO_SCHEMA.md.

Any field can also be set via an ERV2_ITEST_<FIELD> environment variable, e.g. ERV2_ITEST_OUTPUT_DIR=/tmp/out.

CLI flags

Run erv2-itest --help for the always-up-to-date list.

Manually re-running a single step (e.g. cleanup after a crash)

If a run gets interrupted or crashes mid-way, real AWS resources can be left behind without ever reaching the scenario's cleanup step. -k/--select's ::step-name suffix lets you run just that one step - and --run-id lets you target the exact same resources the crashed run created (its identifier is baked into base_input via {{run_id}}, so a fresh random run_id won't find them):

# find the crashed run's ID - it's the directory name under .erv2-itests/runs/
ls .erv2-itests/runs/

uv run erv2-itest integration-tests/my-scenario.yaml \
  --run-id erv2it-20260916120144-c747 -k "::cleanup"

This skips every other step and does not run the scenario's automatic cleanup afterward - it runs exactly the one named step.

🛠️ Development

make dev-env    # uv sync
make format     # ruff check --fix + ruff format
make test       # ruff check, ruff format --check, mypy, pytest --cov
make build      # build the test container image locally

See AGENTS.md for repo layout and the safety notes you should read before running a scenario for real.

📄 License

Apache License 2.0