regrun

Deterministic YAML-driven regression test runner for APIs, MCP servers, and WebSocket streams.

What is regrun?

regrun lets you define regression tests as YAML files and run them against live services — no test framework required. You describe what to call, what to assert, and what to capture; regrun handles execution, variable interpolation, and reporting. It supports four runners: REST APIs (httpx), MCP tools (fastmcp CLI), shell commands (bash), and WebSocket streams (websocket). Tests share a variable store across files, so a JWT captured in setup is available to every subsequent test without any wiring.

Installation

pip install regrun

Requires Python 3.11 or later.

The MCP runner requires uvx and the fastmcp CLI available on PATH:

pip install fastmcp

Quick Start

Create two test files for a fictional myapp running at http://localhost:8000.

tests/regression/00_setup.yaml — acquire a JWT:

meta:
  product: myapp
  layer: setup
  runner: httpx
  endpoint: "http://localhost:8000"

variables:
  RUN_ID: "{{timestamp}}"
  TEST_EMAIL: "regtest-{{RUN_ID}}@example.com"
  TEST_PASSWORD: "TestPass123!"

groups:
  - id: 1
    name: "Auth"
    priority: high
    tests:
      - id: "S.1"
        name: "Login and capture JWT"
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
          json_path:
            "$.access_token": { exists: true }
        capture:
          APP_JWT: "$.access_token"

tests/regression/01_api.yaml — exercise the API with the captured token:

meta:
  product: myapp
  layer: api
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"

groups:
  - id: 1
    name: "Items"
    priority: high
    tests:
      - id: "A.1"
        name: "List items returns array"
        method: GET
        path: "/api/v1/items"
        assert:
          status: 200
          json_path:
            "$": { not_empty: true }

Run the tests:

regrun run tests/regression/

Expected output:

tests/regression/  •  2 tests

  [PASS]  S.1  Login and capture JWT         (142ms)
  [PASS]  A.1  List items returns array       (38ms)

  2 passed, 0 failed  •  180ms

Failure Diagnostics & Run Artifacts

One run tells you everything about a failure — by default. No flag needed. When a test fails or errors, regrun renders a Failures section between the results table and the summary, so Result: PASS|FAIL stays the last line while a tail-clipped terminal still shows the diagnostics:

Failures (1)
============================================================

[Tools] A9.1  List tools
  request: GET http://api:8000/tools
  headers: {'Authorization': '[REDACTED]', 'Content-Type': 'application/json'}
  response status: 500
  response body: {"detail": "RecursionError: maximum recursion depth exceeded"}
  ✗ status: Status 500 != 200
      expected: 200
      actual:   500

------------------------------------------------------------
  Total: 1  Passed: 0  Failed: 1  ...
  Result: FAIL

Full report: /Users/you/.regrun/runs/myproduct/20260711-161301/report.txt (json: report.json)

Each failed test's diagnostics carries the request echo (method/URL/headers/body for httpx, tool+args for MCP, the rendered command list for bash, url/send/wait_for for WebSocket), the response status + body, every failed assertion at full length (not truncated), and the eventually attempt count. Passing tests stay terse. --output json includes diagnostics as an additive field (omitted when null).

Redaction: request headers are redacted at capture time by canonical sensitive-field patterns (authorization, *token*, *key*, cookie, …), and any resolved auth-token value is scrubbed wherever it appears (including a token echoed back in a response body). Response bodies are truncated to 2000 chars (see REGRUN_DIAG_BODY_LIMIT) with a …[truncated, N total chars] annotation.

Persistent artifacts: every run — pass, fail, or --fail-fast abort — writes the full report.txt + report.json + junit.xml to {REGRUN_RUNS_DIR or ~/.regrun/runs}/{product}/{YYYYMMDD-HHMMSS}/, and stdout ends with the pointer line above. An AI agent (or a human) reads the file instead of re-running a multi-minute suite to see a truncated error. Timestamped dirs are never auto-pruned (plain text, negligible size).

JUnit XML for GitLab Tests tab: The junit.xml artifact follows the JUnit spec as consumed by GitLab — one <testsuite> per source YAML file, one <testcase> per test. Failed tests carry a <failure> element with the full diagnostics body (same redaction as report.txt), errored tests <error>, skipped tests <skipped/>. Bodies are capped at 16 KB. Wire it in your .gitlab-ci.yml:

artifacts:
  when: always
  reports:
    junit: regrun-runs/**/junit.xml
  paths:
    - regrun-runs/

How It Works (Execution Model)

File ordering: The setup layer always runs first. All other files run alphabetically by filename. Numeric prefixes (00_, 01_, 02_) enforce the intended order.

Setup dependency: When you pass --layer api or --layer mcp, the setup file is auto-included and runs before the target layer. When setup runs as a dependency, --group and --priority filters are not applied to it — it always runs in full so captured variables stay available. Filters apply to setup only when it is the explicit target (--layer setup). Skip setup entirely with --skip-setup when variables are already populated from a prior run segment.

Cleanup dependency (sweep-first): A group flagged cleanup: true is the mirror of the setup layer on the teardown side. It is always retained under --group / --priority filters (so filtered iteration runs still sweep), and it still executes when --fail-fast aborts the run — in the failing file and every later file — while all other remaining tests are skipped. The run's exit code still reflects the original failure. Suppress cleanup groups with --skip-cleanup when iterating locally. Because within-run cleanup can never be guaranteed (a SIGKILL or crashed run defeats any teardown), the durable pattern is a pattern-based, capture-independent sweep at the start of the run (in 00_setup) that deletes all prior-run artifacts — the run that needs a clean environment is the one that sweeps it. Only such capture-independent sweeps should be flagged cleanup: true.

Variable persistence: File-level variables are merged once per file at parse time. A variable already set by an earlier file — for example RUN_ID defined in setup — is never overwritten by a later file's variables block. This ensures identifiers stay consistent across the entire run.

Layer concept: Tests are organised into four layers, processed in this order:

CLI Reference

regrun run TEST_DIR [OPTIONS]

TEST_DIR is a path to a directory containing YAML test files.

Examples:

# Smoke test only
regrun run tests/regression/ --priority high

# MCP layer only
regrun run tests/regression/ --layer mcp

# Specific groups as JSON
regrun run tests/regression/ --group 1,3 --output json

# Preview without running
regrun run tests/regression/ --dry-run

# Iterate on group 5 without running the sweep groups
regrun run tests/regression/ --group 5 --skip-cleanup

sql Runner

A first-class runner for Postgres statements. It absorbs the psql half of the fleet's hand-rolled bash steps and resolves the docker-exec-vs-direct-psql dispatch once, in Python — no more copy-pasting the command -v docker && docker info guard into every suite. No new DB driver is added: the runner shells out to psql exactly as the bash steps did.

Declare a connection in meta.sql_connection and put SQL in a test's sql: field:

meta:
  product: rally
  layer: api
  runner: sql
  sql_connection:
    docker_container: "{{ env.get('RALLY_COMPOSE_PROJECT', 'rally') }}-db-1"
    docker_user: postgres
    database: "{{ env.get('RALLY_DB', 'rally_prod') }}"
    fallback_dsn: "{{ env.get('RALLY_DSN', 'postgres://postgres@localhost:5432/rally_prod') }}"

groups:
  - id: 5
    name: DB invariants
    tests:
      - id: SQL.1
        name: no orphaned rows
        sql: "SELECT to_jsonb(count(*)) FROM events WHERE org_id IS NULL;"
        assert:
          last_exit_code: 0
          contains: "0"

Dispatch: the runner probes shutil.which("docker") + docker info (cached per run). When docker is available it runs docker exec -i {container} psql -U {user} -d {db}; otherwise psql {fallback_dsn}. Every invocation carries -v ON_ERROR_STOP=1 -q -t -A and receives the statement on stdin. Stdout is parsed JSON-or-string exactly like the bash runner, so contains / json_path on to_jsonb(...) output transfer 1:1.

• Connection values are Jinja-renderable strings — keep the product-prefixed env convention ({{ env.get('RALLY_DB', ...) }}); there are no new REGRUN_SQL_* vars.
• SQL only. App-command steps (docker compose exec … python -m … seeders/reindexers) and OpenSearch curl steps stay runner: bash.
• Adoption is pin-gated: runner: sql hard-fails to parse on a pre-0.8.0 binary (Literal enforcement). A suite may adopt it only after its CI pin is ≥ 0.8.0.

preflight: Dependency-Health Checks

A top-level preflight: block lists read-only probes that run once, before any group, and abort the whole run in seconds naming the failed dependency — killing the degraded-backend grind regime (a slow/broken backend otherwise burns the full suite budget retrying).

meta:
  product: rally
  layer: api
  runner: httpx
  endpoint: http://rally-api:8000

preflight:
  - name: api-reachable
    runner: httpx
    method: GET
    path: /health
    assert:
      status: 200
  - name: db-reachable
    runner: sql
    sql: "SELECT 1;"
    assert:
      last_exit_code: 0

groups:
  - id: 5
    name: API surface
    tests: [...]

• Checks carry a Test-shaped body on any runner, plus a name (used in the abort message) and a timeout (default 10.0s).
• Constraints (validation-enforced): a check may not use eventually: or capture: — a health probe must not retry a degraded backend into looking healthy, nor feed run state.
• Checks are collected across all loaded files in file order. The first failure aborts: the CLI prints PREFLIGHT FAILED: <name> + diagnostics and exits non-zero having executed zero groups.
• --skip-preflight bypasses them (deliberate local override); --dry-run lists them.
• On a passing run the report header prints preflight: N checks passed.
• Compat: preflight: is silently ignored by a pre-0.8.0 binary (unknown key). A CI log missing the preflight: header line was run by an old pin — lint W006 and the header count make this detectable while pins lag.

Per-Product Run Lock

Every run holds an exclusive fcntl.flock on {REGRUN_RUNS_DIR|~/.regrun/runs}/{product}/.lock for its duration, mechanically enforcing the sweep-first no-concurrency assumption. A second concurrent run for the same product exits code 2 naming the product + lock path:

Another regression run for 'rally' is in progress (lock: /…/.regrun/runs/rally/.lock)

• flock self-releases on process death (incl. SIGKILL) — no stale-lock protocol.
• --no-lock bypasses the lock (allow a deliberate concurrent run).
• Network filesystems: flock semantics are unreliable over NFS/CIFS. REGRUN_RUNS_DIR is expected to be local (~/.regrun or the CI workspace).

regrun lint

Static analysis of a suite — no network, no execution. Encodes the regression-testing discipline (assertion strength, budget floors, sweep hygiene, the auth: none trap) as mechanical checks so violations surface at commit time instead of months later as flakes. Exit code is 1 if any error rule fires (0 otherwise); --strict elevates warnings to errors.

regrun lint TARGET [OPTIONS]

TARGET is a directory path or a product name from regrun.yaml.

# Lint before committing suite changes
regrun lint tests/regression/

# CI gate: fail on any warning too
regrun lint tests/regression/ --strict

YAML Schema Reference

meta block (required)

meta:
  product: myapp              # Used for reporting only — does not need to match any registered name
  layer: api                  # setup | api | mcp | chat
  runner: httpx               # httpx | fastmcp | bash | websocket
  endpoint: "http://localhost:8000"      # Base URL for httpx runner
  mcp_endpoint: "http://localhost:9000"  # MCP base URL — falls back to endpoint if omitted
  default_auth: prod          # Auth key applied to all tests without explicit auth:
  env_file: ".env.test"       # Path to .env file, relative to the test file's directory

The product field appears in report output. It does not need to match any external registry.

variables block

variables:
  RUN_ID: "{{timestamp}}"               # Unix timestamp + 4 hex chars (unique per run)
  TODAY: "{{date}}"                     # YYYY-MM-DD
  REQUEST_ID: "{{uuid}}"               # UUID4
  API_TOKEN: "{{env.MY_SECRET_TOKEN}}"  # Environment variable passthrough
  BASE_EMAIL: "admin@myapp.io"          # Static value

Built-in variables:

Full Jinja2 template syntax is supported. The engine runs in StrictUndefined mode: an undefined variable logs a warning and returns the raw template string rather than raising an exception.

Variables set by earlier files are preserved. Downstream files skip re-initialization of keys that already exist in the store.

auth block

auth:
  prod:
    type: bearer                # bearer | api_key
    token: "{{APP_JWT}}"
    org_header: "myapp"         # Sets X-Org-Slug header — omit if not needed
  service_key:
    type: api_key
    token: "{{SERVICE_API_KEY}}"

groups block

groups:
  - id: 1
    name: "Auth Flow"
    priority: high          # high | medium | low  (default: medium)
    context: prod           # prod | fresh | both  (default: prod)
    tests:
      - ...
  - id: 2
    name: "CRUD Operations"
    priority: medium
    tests:
      - ...
  - id: 3
    name: "Environment Sweep"
    cleanup: true           # survives filters; runs even on --fail-fast abort (default: false)
    tests:
      - ...

cleanup: true marks a group as a teardown/sweep that must run even on filtered or aborted runs (see Cleanup dependency above). Reserve it for capture-independent, pattern-based sweeps only — regrun lint flags (W005) a cleanup group that depends on variables captured elsewhere.

Test fields by runner

httpx (REST API)

- id: "A.2"
  name: "Create item"
  method: POST
  path: "/api/v1/items"
  auth: prod                  # Named auth key, "none", or omit to use default_auth
  org_header: true            # false to suppress X-Org-Slug
  body:
    name: "Widget {{RUN_ID}}"
    price: 9.99
  query_params:
    expand: metadata
  assert:
    status: 201
    json_path:
      "$.id": { exists: true }
      "$.name": { starts_with: "Widget" }
  capture:
    ITEM_ID: "$.id"

fastmcp (MCP tools)

- id: "M.1"
  name: "List items via MCP"
  tool: items_list
  args:
    status: "active"
    limit: 10
  auth: service_key
  assert:
    is_error: false
    json_path:
      "$[0].id": { exists: true }
      "$": { not_empty: true }
  capture:
    FIRST_ITEM_ID: "$[0].id"

bash (shell commands)

- id: "S.2"
  name: "Seed test user"
  runner: bash
  commands:
    - cmd: |
        docker exec myapp-postgres psql -U postgres -d myapp \
          -c "INSERT INTO users (email) VALUES ('seed@example.com') ON CONFLICT DO NOTHING;"
      capture:
        RAW_OUTPUT: stdout
  assert:
    last_exit_code: 0
    contains: "INSERT"

Bash commands run from the directory where you invoke regrun, not from the test file location. Use absolute paths or docker exec rather than relative paths.

websocket (streaming)

- id: "C.1"
  name: "Chat session produces response"
  url: "ws://localhost:8000/api/v1/ws/chat?session_id={{SESSION_ID}}"
  send:
    message: "What is the status of my account?"
    session_id: "{{SESSION_ID}}"
  wait_for: "agent_completed"      # Event type that terminates collection
  timeout: 60000                   # Milliseconds (overrides file-level timeout)
  ws_config:
    text_event: text_delta         # Override only if your server uses non-default field names
  assert:
    has_error: false
    json_path:
      "$.response_text": { not_empty: true }
      "$.event_count": { gt: 1 }
  capture:
    CHAT_RESPONSE: "$.response_text"

The runner connects, sends send as a JSON frame, collects events until wait_for is received, and returns an aggregated result dict:

ws_config options (all have defaults — omit unless overriding):

Per-test runner override — used in setup files that mix bash and httpx:

# In a file with meta.runner: bash, a single test can use httpx instead:
- id: "P.1"
  runner: httpx               # Overrides the file-level meta.runner
  method: POST
  path: "/api/v1/auth/login"
  auth: none
  org_header: false
  body:
    email: "{{TEST_EMAIL}}"
    password: "{{TEST_PASSWORD}}"
  assert:
    status: 200
  capture:
    APP_JWT: "$.access_token"

Pure api or mcp files should not use per-test runner: overrides — the file's meta.runner applies uniformly.

Assertion Vocabulary

Top-level assertions

json_path operators

Each entry under json_path: maps a JSONPath expression to one operator:

Note: numeric operators (gt, gte, lt, lte) are the correct names. greater_than, less_than, >=, and <= are not valid.

Response normalization (fastmcp runner)

Assertions and captures run against the normalized tool response body, not the raw MCP payload. Know the shape before writing json_path:

Captures follow the same rule — a capture: path written against the pre-normalize shape (e.g. $.data.id when data gets hoisted) silently captures nothing instead of erroring.

Source: src/regrun/runners/mcp_response.py::_detect_and_normalize (envelope hoist), fastmcp_runner.py::_embed_is_error (string-body wrap).

Variable Capture

capture:
  ITEM_ID: "$.id"                # JSONPath from JSON response
  OWNER_EMAIL: "$.owner.email"   # Nested path
  RAW_OUTPUT: stdout             # Full stdout (bash runner only)

Captured variables are stored in the shared VariableStore and are available to all subsequent tests in the run — including tests in later YAML files. This is how a JWT captured in 00_setup.yaml is accessible in 01_api_surface.yaml without any re-declaration.

Collision avoidance: suffix resource names with {{RUN_ID}} to prevent conflicts across runs:

body:
  name: "Test item {{RUN_ID}}"

Auth Patterns Guide

auth: none is a string literal, not YAML null. Always write auth: none explicitly — writing auth: with no value parses as null and fails.

Multi-file auth flow: setup acquires credentials, downstream files consume them.

00_setup.yaml:

meta:
  runner: httpx
  endpoint: "http://localhost:8000"
# No default_auth — login endpoint needs no auth

groups:
  - id: 1
    tests:
      - id: "S.1"
        name: "Login"
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
        capture:
          APP_JWT: "$.access_token"

01_api_surface.yaml:

meta:
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod          # APP_JWT now available from setup

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"      # Captured in 00_setup.yaml
    org_header: "myapp"

Complete Example

A self-contained two-file example for a fictional myapp REST service.

tests/regression/00_setup.yaml

meta:
  product: myapp
  layer: setup
  runner: bash
  endpoint: "http://localhost:8000"

variables:
  RUN_ID: "{{timestamp}}"
  TEST_EMAIL: "regtest-{{RUN_ID}}@example.com"
  TEST_PASSWORD: "TestPass123!"

groups:
  - id: 1
    name: "Seed"
    priority: high
    tests:
      - id: "S.1"
        name: "Verify database is ready"
        runner: bash
        commands:
          - cmd: "docker exec myapp-postgres pg_isready -U postgres"
            capture:
              RAW_OUTPUT: stdout
        assert:
          last_exit_code: 0
          contains: "accepting connections"

      - id: "S.2"
        name: "Login and capture JWT"
        runner: httpx
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
          json_path:
            "$.access_token": { exists: true }
        capture:
          APP_JWT: "$.access_token"

      - id: "S.3"
        name: "Create API key"
        runner: httpx
        method: POST
        path: "/api/v1/api-keys"
        auth: session
        body:
          name: "regression-key-{{RUN_ID}}"
        assert:
          status: 201
          json_path:
            "$.key": { starts_with: "ak_" }
        capture:
          API_KEY: "$.key"

auth:
  session:
    type: bearer
    token: "{{APP_JWT}}"

tests/regression/01_api_surface.yaml

meta:
  product: myapp
  layer: api
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"
    org_header: "myapp"

groups:
  - id: 1
    name: "Items CRUD"
    priority: high
    tests:
      - id: "A.1"
        name: "List items"
        method: GET
        path: "/api/v1/items"
        assert:
          status: 200
          json_path:
            "$": { not_empty: true }

      - id: "A.2"
        name: "Create item"
        method: POST
        path: "/api/v1/items"
        body:
          name: "Regression item {{RUN_ID}}"
          price: 19.99
        assert:
          status: 201
          json_path:
            "$.id": { exists: true }
            "$.name": { contains: "Regression item" }
        capture:
          ITEM_ID: "$.id"

      - id: "A.3"
        name: "Get item by ID"
        method: GET
        path: "/api/v1/items/{{ITEM_ID}}"
        assert:
          status: 200
          json_path:
            "$.id": { equals: "{{ITEM_ID}}" }
            "$.price": { equals: "19.99" }

      - id: "A.4"
        name: "Delete item"
        method: DELETE
        path: "/api/v1/items/{{ITEM_ID}}"
        assert:
          status: 204

Run it:

regrun run tests/regression/

Environment Variables

CI Integration

In CI, services run as Docker containers with network aliases instead of *.localhost domains. Use the endpoint override variables to point regrun at the container aliases.

GitLab CI:

regression:
  stage: test
  services:
    - name: myapp-api:latest
      alias: api
    - name: myapp-mcp:latest
      alias: mcp
  variables:
    REGRUN_API_ENDPOINT: "http://api:8000"
    REGRUN_MCP_ENDPOINT: "http://mcp:9000"
  script:
    - pip install regrun
    - regrun run tests/regression/

GitHub Actions:

jobs:
  regression:
    runs-on: ubuntu-latest
    services:
      api:
        image: myapp-api:latest
        ports:
          - 8000:8000
    steps:
      - uses: actions/checkout@v4
      - run: pip install regrun
      - run: regrun run tests/regression/
        env:
          REGRUN_API_ENDPOINT: "http://localhost:8000"

The endpoint override applies to every test file in the run. YAML files keep their local *.localhost URLs for developer use; CI overrides them without any file changes.

File Structure

Recommended test directory layout:

tests/regression/
  00_setup.yaml          # Setup: auth, seed data, environment checks
  01_api_surface.yaml    # REST API surface tests
  02_mcp_surface.yaml    # MCP tool tests
  03_chat_surface.yaml   # WebSocket / streaming tests

Numeric prefixes control alphabetical sort order. The setup layer is always processed first regardless of filename, but 00_ makes the intent explicit and keeps directory listings readable.

Development

Install dependencies and run the test suite:

poetry install
poetry run pytest

Tests live at tests/integration/cli/ and cover CLI behaviour end-to-end.

License

MIT