regrun

Deterministic YAML-driven regression test runner for APIs, MCP servers, and WebSocket streams.

What is regrun?

regrun lets you define regression tests as YAML files and run them against live services, with no test framework required. You describe what to call, what to assert, and what to capture; regrun handles execution, variable interpolation, and reporting. It supports four runners: REST APIs (httpx), MCP tools (fastmcp CLI), shell commands (bash), and WebSocket streams (websocket). Tests share a variable store across files, so a JWT captured in setup is available to every subsequent test without any wiring.

Installation

pip install regrun

Requires Python 3.11 or later.

The MCP runner requires uvx and the fastmcp CLI available on PATH:

pip install fastmcp

Quick Start

Create two test files for a fictional myapp running at http://localhost:8000.

tests/regression/00_setup.yaml acquires a JWT:

meta:
  product: myapp
  layer: setup
  runner: httpx
  endpoint: "http://localhost:8000"

variables:
  RUN_ID: "{{timestamp}}"
  TEST_EMAIL: "regtest-{{RUN_ID}}@example.com"
  TEST_PASSWORD: "TestPass123!"

groups:
  - id: 1
    name: "Auth"
    priority: high
    tests:
      - id: "S.1"
        name: "Login and capture JWT"
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
          json_path:
            "$.access_token": { exists: true }
        capture:
          APP_JWT: "$.access_token"

tests/regression/01_api.yaml exercises the API with the captured token:

meta:
  product: myapp
  layer: api
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"

groups:
  - id: 1
    name: "Items"
    priority: high
    tests:
      - id: "A.1"
        name: "List items returns array"
        method: GET
        path: "/api/v1/items"
        assert:
          status: 200
          json_path:
            "$": { not_empty: true }

Run the tests:

regrun run tests/regression/

Expected output:

tests/regression/  •  2 tests

  [PASS]  S.1  Login and capture JWT         (142ms)
  [PASS]  A.1  List items returns array       (38ms)

  2 passed, 0 failed  •  180ms

Failure Diagnostics & Run Artifacts

One run tells you everything about a failure, by default. No flag needed. When a test fails or errors, regrun renders a Failures section between the results table and the summary, so Result: PASS|FAIL stays the last line while a tail-clipped terminal still shows the diagnostics:

Failures (1)
============================================================

[Tools] A9.1  List tools
  request: GET http://api:8000/tools
  headers: {'Authorization': '[REDACTED]', 'Content-Type': 'application/json'}
  response status: 500
  response body: {"detail": "RecursionError: maximum recursion depth exceeded"}
  ✗ status: Status 500 != 200
      expected: 200
      actual:   500

------------------------------------------------------------
  Total: 1  Passed: 0  Failed: 1  ...
  Result: FAIL

Full report: /Users/you/.regrun/runs/myapp/20260711-161301/report.txt (json: report.json)

Each failed test's diagnostics carries the request echo (method/URL/headers/body for httpx, tool+args for MCP, the rendered command list for bash, url/send/wait_for for WebSocket), the response status + body, every failed assertion at full length (not truncated), and the eventually attempt count. Passing tests stay terse. --output json includes diagnostics as an additive field (omitted when null).

Redaction: request headers are redacted at capture time by canonical sensitive-field patterns (authorization, *token*, *key*, cookie, …), and any resolved auth-token value is scrubbed wherever it appears (including a token echoed back in a response body). Response bodies are truncated to 2000 chars (see REGRUN_DIAG_BODY_LIMIT) with a …[truncated, N total chars] annotation.

Persistent artifacts: every run (pass, fail, or --fail-fast abort) writes the full report.txt + report.json + junit.xml to {REGRUN_RUNS_DIR or ~/.regrun/runs}/{product}/{YYYYMMDD-HHMMSS}/, and stdout ends with the pointer line above. An AI agent (or a human) reads the file instead of re-running a multi-minute suite to see a truncated error. Timestamped dirs are never auto-pruned (plain text, negligible size).

JUnit XML for GitLab Tests tab: The junit.xml artifact follows the JUnit spec as consumed by GitLab: one <testsuite> per source YAML file, one <testcase> per test. Failed tests carry a <failure> element with the full diagnostics body (same redaction as report.txt), errored tests <error>, skipped tests <skipped/>. Bodies are capped at 16 KB. Wire it in your .gitlab-ci.yml:

artifacts:
  when: always
  reports:
    junit: regrun-runs/**/junit.xml
  paths:
    - regrun-runs/

How It Works (Execution Model)

File ordering: One order governs the whole engine: layer rank first (setup, api, mcp, chat; an unrecognised layer runs last), then the file's stem in byte order. Numeric prefixes (00_, 01_, 02_) enforce the intended order. The runner, the --file and --rerun-failed selectors, the shard planner and the linter's ordering rules all read that one definition, so a plan, a report and a lint finding can never disagree about which file comes first.

The stem is the filename without .yaml, and excluding the extension is deliberate: it is a constant that carries no ordering intent, and comparing it against real characters is what makes punctuated names surprising. Given two files in one layer:

Names built from digits, letters and underscores are unaffected by any of this, which is what the numeric-prefix convention is for.

Upgrading from 0.9.x can reorder files within a layer. 0.9.x ordered by the full filename, extension included. The order moves for any pair whose stem is a byte-for-byte prefix of another stem that continues with a character sorting below . (a hyphen is the common one), so 00_setup and 00_setup-extra swap. Run regrun run <dir> --dry-run and read the file list before upgrading a pinned CI suite. Lint rule E002 compares by the same key, so its verdict can flip for such a pair: a suite holding cleanup file 10_cleanup.yaml next to mcp file 10_cleanup-extra.yaml was not flagged before and is flagged now, correctly, because that mcp file does run after the cleanup file. Renaming either file clears it.

Setup dependency: When you pass --layer api or --layer mcp, the setup file is auto-included and runs before the target layer. When setup runs as a dependency, --group and --priority filters are not applied to it: it always runs in full so captured variables stay available. Filters apply to setup only when it is the explicit target (--layer setup). Skip setup entirely with --skip-setup when variables are already populated from a prior run segment.

Selecting a setup file (--file): The setup layer is ordered and single-homed: the first setup file owns the suite's variables: and meta.env_file, and any setup file after it may read them. So selecting a setup-layer file also runs every setup file sorting before it (plus their own requires closures). A setup file sorting after the selection is never pulled: nothing it produces can have existed when the selected file ran in a full suite, so needing it would be a suite defect rather than a dependency. Selecting the first setup file therefore runs that file alone, and --skip-setup still removes the whole layer, which makes a --file pattern that only matched setup files an error, never a silent zero-file run.

Cleanup dependency (sweep-first): A group flagged cleanup: true is the mirror of the setup layer on the teardown side. It is always retained under --group / --priority filters (so filtered iteration runs still sweep), and it still executes when --fail-fast aborts the run, in the failing file and every later file, while all other remaining tests are skipped. The run's exit code still reflects the original failure. Suppress cleanup groups with --skip-cleanup when iterating locally. Because within-run cleanup can never be guaranteed (a SIGKILL or crashed run defeats any teardown), the durable pattern is a pattern-based, capture-independent sweep at the start of the run (in 00_setup) that deletes all prior-run artifacts. The run that needs a clean environment is the one that sweeps it. Only such capture-independent sweeps should be flagged cleanup: true.

Declared file dependencies: A file lists the files it consumes captured values from in meta.requires: (stems, no .yaml). The setup layer is the bootstrap contract every file already depends on and is never listed. Declaring a dependency buys three things: the file selectors pull the producer in automatically, sharding keeps a file and its closure together, and a failed producer reports its consumers as BLOCKED. Lint rule W012 finds the couplings a suite has not declared yet.

A requires: entry that names no file in the suite directory, names the file itself, or closes a cycle aborts the run before anything executes: there is no order that satisfies it, and dropping it silently would leave the run green with blocked-skip quietly disabled. A producer left out by your own narrowing (--file, --layer, --shard) is the one tolerated case, because a filtered run cannot judge a producer it never loaded.

BLOCKED: When a file fails, every file that requires it (directly or transitively) is not run, and its tests are reported BLOCKED naming the file that blocked them. A failed layer: setup file blocks every later file, declared dependency or not: setup is the bootstrap contract nobody declares, so a seed file reporting green after a dead auth bootstrap would send the reader to the wrong place. Blocked tests are a sub-kind of skipped: one broken producer yields one actionable failure instead of a cascade of red that all has the same cause. The exit code is driven by real failures and errors, so a run whose only red is a blocked consumer still points at the producer.

Variable persistence: File-level variables are merged once per file at parse time. A variable already set by an earlier file (for example RUN_ID defined in setup) is never overwritten by a later file's variables block. This ensures identifiers stay consistent across the entire run.

Layer concept: Tests are organised into four layers, processed in this order:

CLI Reference

regrun run TEST_DIR [OPTIONS]

TEST_DIR is a path to a directory containing YAML test files.

Time budgets are off unless declared. A file declares its own ceiling with meta.budget_seconds; --budget-seconds covers the whole run. An overrun reds the run and the report names the file and the overrun, but it never reclassifies a test: the test passed, the budget did not.

Sharding requires disjoint environments. Each shard needs its own database and its own search-index prefix. Two shards against one stack overwrite each other's fixtures and both verdicts become meaningless. regrun cannot verify the precondition, so it is the caller's to honour. Shards are built from the dependency graph: a file and its requires closure always land in the same shard, the setup layer runs in every shard, and a serial: true file gets the last shard to itself. Balance comes from greedy longest-processing-time packing, by test count when no timings are supplied. Plans are deterministic, so --dry-run --shard k/n can be diffed before a matrix is wired.

Narrowing does not narrow validation. --file, --rerun-failed and --shard are applied after every discovered file has been parsed and validated, because selection reads what the files declare (the requires closure, the shard weights, the run order). So one schema-invalid file aborts every run of that directory, including a --file run that did not select it, and including --dry-run. That is intended: a suite holding a file the engine cannot load is not a suite a narrowed green can be trusted from, and the rest of the directory is the context the selection was computed in. (--layer and --skip-setup are the exception, because they narrow file discovery rather than the selection, so files they exclude are never read.)

To recover, lint the directory: regrun lint <dir> names the file and the exact key path as E006, for every offending file at once rather than one abort at a time. Fix the keys it reports, then re-run the narrowed command.

Examples:

# Smoke test only
regrun run tests/regression/ --priority high

# MCP layer only
regrun run tests/regression/ --layer mcp

# Specific groups as JSON
regrun run tests/regression/ --group 1,3 --output json

# Preview without running
regrun run tests/regression/ --dry-run

# Iterate on group 5 without running the sweep groups
regrun run tests/regression/ --group 5 --skip-cleanup

# One file, with setup and everything it requires
regrun run tests/regression/ --file 11_search_e2e

# Every file of one family (glob), repeatable
regrun run tests/regression/ --file "02_mcp_*" --file 05_flows

# One setup file, with the setup files that run before it
regrun run tests/regression/ --file 00g_seed_directory

# After a red run: re-run only what broke
regrun run tests/regression/ --rerun-failed

# Shard 2 of 3, against that shard's own stack
regrun run tests/regression/ --shard 2/3

# Diff the planned subsets before wiring a CI matrix
regrun run tests/regression/ --dry-run --shard 1/3

sql Runner

A first-class runner for Postgres statements. It absorbs the psql half of the fleet's hand-rolled bash steps and resolves the docker-exec-vs-direct-psql dispatch once, in Python: no more copy-pasting the command -v docker && docker info guard into every suite. No new DB driver is added: the runner shells out to psql exactly as the bash steps did.

Declare a connection in meta.sql_connection and put SQL in a test's sql: field:

meta:
  product: myapp
  layer: api
  runner: sql
  sql_connection:
    docker_container: "{{ env.get('MYAPP_COMPOSE_PROJECT', 'myapp') }}-db-1"
    docker_user: postgres
    database: "{{ env.get('MYAPP_DB', 'myapp_test') }}"
    fallback_dsn: "{{ env.get('MYAPP_DSN', 'postgres://postgres@localhost:5432/myapp_test') }}"

groups:
  - id: 5
    name: DB invariants
    tests:
      - id: SQL.1
        name: no orphaned rows
        sql: "SELECT to_jsonb(count(*)) FROM events WHERE org_id IS NULL;"
        assert:
          last_exit_code: 0
          contains: "0"

Dispatch: the runner probes shutil.which("docker") + docker info (cached per run). When docker is available it runs docker exec -i {container} psql -U {user} -d {db}; otherwise psql {fallback_dsn}. Every invocation carries -v ON_ERROR_STOP=1 -q -t -A and receives the statement on stdin. Stdout is parsed JSON-or-string exactly like the bash runner, so contains / json_path on to_jsonb(...) output transfer 1:1.

• Connection values are Jinja-renderable strings. Keep the product-prefixed env convention ({{ env.get('MYAPP_DB', ...) }}); there are no new REGRUN_SQL_* vars.
• SQL only. App-command steps (docker compose exec … python -m … seeders/reindexers) and OpenSearch curl steps stay runner: bash.
• Adoption is pin-gated: runner: sql hard-fails to parse on a pre-0.8.0 binary (Literal enforcement). A suite may adopt it only after its CI pin is ≥ 0.8.0.

preflight: Dependency-Health Checks

A top-level preflight: block lists read-only probes that run once, before any group, and abort the whole run in seconds naming the failed dependency. This kills the degraded-backend grind regime (a slow/broken backend otherwise burns the full suite budget retrying).

meta:
  product: myapp
  layer: api
  runner: httpx
  endpoint: http://myapp-api:8000

preflight:
  - name: api-reachable
    runner: httpx
    method: GET
    path: /health
    assert:
      status: 200
  - name: db-reachable
    runner: sql
    sql: "SELECT 1;"
    assert:
      last_exit_code: 0

groups:
  - id: 5
    name: API surface
    tests: [...]

• Checks carry a Test-shaped body on any runner, plus a name (used in the abort message) and a timeout (default 10.0s).
• Constraints (validation-enforced): a check may not use eventually: or capture:. A health probe must not retry a degraded backend into looking healthy, nor feed run state.
• Checks are collected across all loaded files in file order. The first failure aborts: the CLI prints PREFLIGHT FAILED: <name> + diagnostics and exits non-zero having executed zero groups.
• --skip-preflight bypasses them (deliberate local override); --dry-run lists them.
• On a passing run the report header prints preflight: N checks passed.
• Compat: preflight: is silently ignored by a pre-0.8.0 binary (unknown key). A CI log missing the preflight: header line was run by an old pin. Lint W006 and the header count make this detectable while pins lag.

Per-Product Run Lock

Every run holds an exclusive fcntl.flock on {REGRUN_RUNS_DIR|~/.regrun/runs}/{product}/.lock for its duration, mechanically enforcing the sweep-first no-concurrency assumption. A second concurrent run for the same product exits code 2 naming the product + lock path:

Another regression run for 'myapp' is in progress (lock: /…/.regrun/runs/myapp/.lock)

• flock self-releases on process death (incl. SIGKILL), so there is no stale-lock protocol.
• --no-lock bypasses the lock (allow a deliberate concurrent run).
• Network filesystems: flock semantics are unreliable over NFS/CIFS. REGRUN_RUNS_DIR is expected to be local (~/.regrun or the CI workspace).

regrun lint

Static analysis of a suite: no network, no execution. Encodes the regression-testing discipline (assertion strength, budget floors, sweep hygiene, the auth: none trap) as mechanical checks so violations surface at commit time instead of months later as flakes. Exit code is 1 if any error rule fires (0 otherwise); --strict elevates warnings to errors.

regrun lint TARGET [OPTIONS]

TARGET is a directory path or a product name from regrun.yaml.

# Lint before committing suite changes
regrun lint tests/regression/

# CI gate: fail on any warning too
regrun lint tests/regression/ --strict

YAML Schema Reference

meta block (required)

meta:
  product: myapp              # Used for reporting only; does not need to match any registered name
  layer: api                  # setup | api | mcp | chat
  runner: httpx               # httpx | fastmcp | bash | websocket
  endpoint: "http://localhost:8000"      # Base URL for httpx runner
  mcp_endpoint: "http://localhost:9000"  # MCP base URL; falls back to endpoint if omitted
  default_auth: prod          # Auth key applied to all tests without explicit auth:
  env_file: ".env.test"       # Path to .env file, relative to the test file's directory
  strict_vars: true           # Default true: an unresolved {{VAR}} fails the test
  requires: ["01_provider"]   # Files (stems) this one consumes captured values from
  serial: false               # True: asserts process-global state, never shares a shard
  budget_seconds: 60          # Optional wall-clock ceiling for this file
  health_path: "/health"      # Read by external orchestration only; the engine ignores it
  mcp_health_path: "/mcp/health"

The product field appears in report output. It does not need to match any external registry.

Unknown meta keys are rejected at load. A typo such as require: would otherwise be accepted and silently ignored, leaving the file with no declared dependency and nothing anywhere saying so.

Unknown keys are rejected everywhere

Every block in this reference rejects keys it does not declare: the file itself, meta, an auth profile, preflight and sweep steps, a group, a test, assert, a bash command, eventually, ws_config and meta.sql_connection. A key a block does not declare is read by nothing, so accepting it would mean accepting a condition or a setting that never takes effect. (variables: and a capture: mapping are open by design: their keys are variable names you choose.)

Both entrypoints are loud about it. regrun run aborts before executing anything, naming the file, the location and the key:

Error: Failed to parse 01_items.yaml: 1 validation error for TestFile
groups.0.tests.0.commands.0.assert
  Extra inputs are not permitted [type=extra_forbidden, ...]

regrun lint reports the same thing as error E006, so the static gate cannot pass a file the engine would refuse:

01_items.yaml:A.1 E006 (error) schema violation at groups.0.tests.0.commands.0.assert: Extra inputs are not permitted

There is no opt-out. Read the reported location, then move the key to the block that declares it, or remove it.

variables block

variables:
  RUN_ID: "{{timestamp}}"               # Unix timestamp + 4 hex chars (unique per run)
  TODAY: "{{date}}"                     # YYYY-MM-DD
  REQUEST_ID: "{{uuid}}"               # UUID4
  API_TOKEN: "{{env.MY_SECRET_TOKEN}}"  # Environment variable passthrough
  BASE_EMAIL: "admin@myapp.io"          # Static value

Built-in variables:

Full Jinja2 template syntax is supported. The engine runs in StrictUndefined mode: an undefined variable logs a warning and returns the raw template string rather than raising an exception.

Variables set by earlier files are preserved. Downstream files skip re-initialization of keys that already exist in the store.

auth block

auth:
  prod:
    type: bearer                # bearer | api_key
    token: "{{APP_JWT}}"
    org_header: "myapp"         # Sets X-Org-Slug header; omit if not needed
  service_key:
    type: api_key
    token: "{{SERVICE_API_KEY}}"

groups block

groups:
  - id: 1
    name: "Auth Flow"
    priority: high          # high | medium | low  (default: medium)
    context: prod           # prod | fresh | both  (default: prod)
    tests:
      - ...
  - id: 2
    name: "CRUD Operations"
    priority: medium
    tests:
      - ...
  - id: 3
    name: "Environment Sweep"
    cleanup: true           # survives filters; runs even on --fail-fast abort (default: false)
    tests:
      - ...

cleanup: true marks a group as a teardown/sweep that must run even on filtered or aborted runs (see Cleanup dependency above). Reserve it for capture-independent, pattern-based sweeps only. regrun lint flags (W005) a cleanup group that depends on variables captured elsewhere.

Test fields by runner

httpx (REST API)

- id: "A.2"
  name: "Create item"
  method: POST
  path: "/api/v1/items"
  auth: prod                  # Named auth key, "none", or omit to use default_auth
  org_header: true            # false to suppress X-Org-Slug
  body:
    name: "Widget {{RUN_ID}}"
    price: 9.99
  query_params:
    expand: metadata
  assert:
    status: 201
    json_path:
      "$.id": { exists: true }
      "$.name": { starts_with: "Widget" }
  capture:
    ITEM_ID: "$.id"

fastmcp (MCP tools)

- id: "M.1"
  name: "List items via MCP"
  tool: items_list
  args:
    status: "active"
    limit: 10
  auth: service_key
  assert:
    is_error: false
    json_path:
      "$[0].id": { exists: true }
      "$": { not_empty: true }
  capture:
    FIRST_ITEM_ID: "$[0].id"

bash (shell commands)

- id: "S.2"
  name: "Seed test user"
  runner: bash
  commands:
    - cmd: |
        docker exec myapp-postgres psql -U postgres -d myapp \
          -c "INSERT INTO users (email) VALUES ('seed@example.com') ON CONFLICT DO NOTHING;"
      capture:
        RAW_OUTPUT: stdout
  assert:
    last_exit_code: 0
    contains: "INSERT"

Bash commands run from the directory where you invoke regrun, not from the test file location. Use absolute paths or docker exec rather than relative paths.

websocket (streaming)

- id: "C.1"
  name: "Chat session produces response"
  url: "ws://localhost:8000/api/v1/ws/chat?session_id={{SESSION_ID}}"
  send:
    message: "What is the status of my account?"
    session_id: "{{SESSION_ID}}"
  wait_for: "agent_completed"      # Event type that terminates collection
  timeout: 60000                   # Milliseconds (overrides file-level timeout)
  ws_config:
    text_event: text_delta         # Override only if your server uses non-default field names
  assert:
    has_error: false
    json_path:
      "$.response_text": { not_empty: true }
      "$.event_count": { gt: 1 }
  capture:
    CHAT_RESPONSE: "$.response_text"

The runner connects, sends send as a JSON frame, collects events until wait_for is received, and returns an aggregated result dict:

ws_config options (all have defaults; omit unless overriding):

Per-test runner override, used in setup files that mix bash and httpx:

# In a file with meta.runner: bash, a single test can use httpx instead:
- id: "P.1"
  runner: httpx               # Overrides the file-level meta.runner
  method: POST
  path: "/api/v1/auth/login"
  auth: none
  org_header: false
  body:
    email: "{{TEST_EMAIL}}"
    password: "{{TEST_PASSWORD}}"
  assert:
    status: 200
  capture:
    APP_JWT: "$.access_token"

Pure api or mcp files should not use per-test runner: overrides; the file's meta.runner applies uniformly.

Assertion Vocabulary

Top-level assertions

json_path operators

Each entry under json_path: maps a JSONPath expression to one or more operators. Every operator listed under a path is evaluated and reported separately, and the test passes only when all of them pass (AND):

Note: numeric operators (gt, gte, lt, lte) are the correct names. greater_than, less_than, >=, and <= are not valid.

A path carrying several operators ("$.slug": { not_empty: true, starts_with: "myapp-" }) reports one result per operator, so the report names which one failed. An unrecognised key under a path fails the test rather than being ignored.

Response normalization (fastmcp runner)

Assertions and captures run against the normalized tool response body, not the raw MCP payload. Know the shape before writing json_path:

Captures follow the same rule: a capture: path written against the pre-normalize shape (e.g. $.data.id when data gets hoisted) silently captures nothing instead of erroring.

Source: src/regrun/runners/mcp_response.py::_detect_and_normalize (envelope hoist), fastmcp_runner.py::_embed_is_error (string-body wrap).

Variable Capture

capture:
  ITEM_ID: "$.id"                # JSONPath from JSON response
  OWNER_EMAIL: "$.owner.email"   # Nested path
  RAW_OUTPUT: stdout             # Full stdout (bash runner only)

Captured variables are stored in the shared VariableStore and are available to all subsequent tests in the run, including tests in later YAML files. This is how a JWT captured in 00_setup.yaml is accessible in 01_api_surface.yaml without any re-declaration.

Collision avoidance: suffix resource names with {{RUN_ID}} to prevent conflicts across runs:

body:
  name: "Test item {{RUN_ID}}"

Auth Patterns Guide

auth: none is a string literal, not YAML null. Always write auth: none explicitly. Writing auth: with no value parses as null and fails.

Multi-file auth flow: setup acquires credentials, downstream files consume them.

00_setup.yaml:

meta:
  runner: httpx
  endpoint: "http://localhost:8000"
# No default_auth: login endpoint needs no auth

groups:
  - id: 1
    tests:
      - id: "S.1"
        name: "Login"
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
        capture:
          APP_JWT: "$.access_token"

01_api_surface.yaml:

meta:
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod          # APP_JWT now available from setup

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"      # Captured in 00_setup.yaml
    org_header: "myapp"

Complete Example

A self-contained two-file example for a fictional myapp REST service.

tests/regression/00_setup.yaml

meta:
  product: myapp
  layer: setup
  runner: bash
  endpoint: "http://localhost:8000"

variables:
  RUN_ID: "{{timestamp}}"
  TEST_EMAIL: "regtest-{{RUN_ID}}@example.com"
  TEST_PASSWORD: "TestPass123!"

groups:
  - id: 1
    name: "Seed"
    priority: high
    tests:
      - id: "S.1"
        name: "Verify database is ready"
        runner: bash
        commands:
          - cmd: "docker exec myapp-postgres pg_isready -U postgres"
            capture:
              RAW_OUTPUT: stdout
        assert:
          last_exit_code: 0
          contains: "accepting connections"

      - id: "S.2"
        name: "Login and capture JWT"
        runner: httpx
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
          json_path:
            "$.access_token": { exists: true }
        capture:
          APP_JWT: "$.access_token"

      - id: "S.3"
        name: "Create API key"
        runner: httpx
        method: POST
        path: "/api/v1/api-keys"
        auth: session
        body:
          name: "regression-key-{{RUN_ID}}"
        assert:
          status: 201
          json_path:
            "$.key": { starts_with: "ak_" }
        capture:
          API_KEY: "$.key"

auth:
  session:
    type: bearer
    token: "{{APP_JWT}}"

tests/regression/01_api_surface.yaml

meta:
  product: myapp
  layer: api
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"
    org_header: "myapp"

groups:
  - id: 1
    name: "Items CRUD"
    priority: high
    tests:
      - id: "A.1"
        name: "List items"
        method: GET
        path: "/api/v1/items"
        assert:
          status: 200
          json_path:
            "$": { not_empty: true }

      - id: "A.2"
        name: "Create item"
        method: POST
        path: "/api/v1/items"
        body:
          name: "Regression item {{RUN_ID}}"
          price: 19.99
        assert:
          status: 201
          json_path:
            "$.id": { exists: true }
            "$.name": { contains: "Regression item" }
        capture:
          ITEM_ID: "$.id"

      - id: "A.3"
        name: "Get item by ID"
        method: GET
        path: "/api/v1/items/{{ITEM_ID}}"
        assert:
          status: 200
          json_path:
            "$.id": { equals: "{{ITEM_ID}}" }
            "$.price": { equals: "19.99" }

      - id: "A.4"
        name: "Delete item"
        method: DELETE
        path: "/api/v1/items/{{ITEM_ID}}"
        assert:
          status: 204

Run it:

regrun run tests/regression/

Environment Variables

CI Integration

In CI, services run as Docker containers with network aliases instead of *.localhost domains. Use the endpoint override variables to point regrun at the container aliases.

GitLab CI:

regression:
  stage: test
  services:
    - name: myapp-api:latest
      alias: api
    - name: myapp-mcp:latest
      alias: mcp
  variables:
    REGRUN_API_ENDPOINT: "http://api:8000"
    REGRUN_MCP_ENDPOINT: "http://mcp:9000"
  script:
    - pip install regrun
    - regrun run tests/regression/

GitHub Actions:

jobs:
  regression:
    runs-on: ubuntu-latest
    services:
      api:
        image: myapp-api:latest
        ports:
          - 8000:8000
    steps:
      - uses: actions/checkout@v4
      - run: pip install regrun
      - run: regrun run tests/regression/
        env:
          REGRUN_API_ENDPOINT: "http://localhost:8000"

The endpoint override applies to every test file in the run. YAML files keep their local *.localhost URLs for developer use; CI overrides them without any file changes.

File Structure

Recommended test directory layout:

tests/regression/
  00_setup.yaml          # Setup: auth, seed data, environment checks
  01_api_surface.yaml    # REST API surface tests
  02_mcp_surface.yaml    # MCP tool tests
  03_chat_surface.yaml   # WebSocket / streaming tests

Numeric prefixes control the within-layer sort order (see File ordering above: layer rank, then the stem in byte order). The setup layer is always processed first whatever the names are, but 00_ makes the intent explicit and keeps directory listings readable.

Development

Install dependencies and run the test suite:

poetry install
poetry run pytest

Tests live at tests/integration/cli/ and cover CLI behaviour end-to-end.

License

MIT

Maintained by Neomanex.