xeet

Declarative, language-agnostic test orchestrator driven by YAML and JSON.

xeet is an end-to-end testing framework and test runner. Instead of embedding test logic inside a specific programming language, xeet defines test suites declaratively in configuration files, orchestrating arbitrary commands, scripts, and processes across structured execution phases with automated verification.

---

Key Features

• Language-Agnostic: Orchestrate CLI binaries, test scripts, APIs, or shell commands across any stack.
• Declarative Configuration: Define suites in YAML or JSON with support for modular include files.
• Phased Test Execution:
  • pre_run — Setup steps that stop on failure.
  • run — Main test execution with expected exit codes, timeouts, and output verification.
  • post_run — Cleanup and tear-down steps that execute even if main steps fail.
• Dual Inheritance Model:
  • Test Inheritance: Re-use and extend base test definitions (prepend, append, or replace steps).
  • Step Inheritance: Reference reusable step definitions via JSONPath (settings.common_steps.*).
• Flexible Verification: Match standard output and standard error against strings or expected files, with regex/string scrubbing filters for hermetic diffs.
• Scoped Variable System: Recursive string interpolation ({var}), environment variable access ({$ENV_VAR}), object references ($ref://...), and built-in runtime variables ({XEET_ROOT}, {XEET_OUT_DIR}, etc.).
• Platform-Specific Testing: Target specific OS environments (posix, nt), inherit platform constraints, and load platform-specific config files dynamically via {XEET_PLATFORM}.
• Parallel Execution: Execute tests concurrently across worker threads with -j/--jobs (defaults to auto-detecting core count).
• Resource Pools & Concurrency Control: Define shared resource pools (ports, database instances, accounts) and allocate them to tests to prevent contention or serialize specific tests during parallel execution.
• Parameter Matrix: Define global matrix variables (lists of values) to automatically execute test suites across the full cartesian product of permutations.
• Test Randomization: Shuffle test execution order per iteration with --randomize to expose hidden inter-test dependencies.
• Fine-Grained Filtering: Select tests by exact name, fuzzy match, or tag groups with include/exclude rules.
• Rich Terminal UI: Live progress display with customizable output detail, timing breakdowns, and --debug live process tailing.

---

Installation

Requires Python >= 3.10.

pip install xeet

Or install from source:

git clone https://github.com/omercsp/xeet.git
cd xeet
pip install .

---

Quick Start

Create a configuration file named xeet.yaml:

variables:
  greeting: "Hello, World!"

settings:
  xeet:
    default_step_type: exec

tests:
  - name: hello_world
    short_desc: Verify basic echo command
    groups: [smoke, sanity]
    run:
      - cmd: echo "{greeting}"
        expected_stdout: "Hello, World!\n"

  - name: check_system
    short_desc: Run a system check with environment variables
    groups: [sanity]
    pre_run:
      - cmd: mkdir -p "{XEET_TEST_OUT_DIR}/data"
    run:
      - cmd: python -c "import os; print(os.environ['APP_ENV'])"
        env:
          APP_ENV: "testing"
        expected_stdout: "testing\n"
    post_run:
      - cmd: rm -rf "{XEET_TEST_OUT_DIR}/data"

Run the suite:

xeet run

---

Configuration Guide

A xeet configuration file (xeet.yaml, xeet.yml, or xeet.json) consists of four primary sections:

1. include

Modularize configurations by including other YAML/JSON files. Includes are merged recursively with loop detection:

include:
  - shared_variables.yaml
  - step_library.yaml

2. variables

Define scoped variables for string interpolation ({var_name}) or object references ($ref://var_name):

variables:
  api_host: "http://localhost:8080"
  timeout_sec: 5
  db_config:
    user: "admin"
    port: 5432

• Environment variables: Access system environment via {$ENV_VAR_NAME}.
• Escaping: Use \{ to escape literal braces.
• Built-in System Variables:
  • {XEET_CWD} — Current working directory at launch.
  • {XEET_ROOT} — Directory containing the active configuration file.
  • {XEET_OUT_DIR} — Output directory root for test runs.
  • {XEET_EXPECTED_DIR} — Expected baselines directory root.
  • {XEET_TEST_NAME} — Name of the currently executing test.
  • {XEET_TEST_OUT_DIR} — Output directory dedicated to the current test.
  • {XEET_STEP_OUT_DIR} — Output directory for the current step.
  • {XEET_STEP_INDEX} — Zero-based index of the step within its phase.
  • {XEET_ITERATIONS} — Total iteration count (-r flag).
  • {XEET_DEBUG} — Set to 1 when --debug is active, otherwise 0.
  • {XEET_PLATFORM} — Operating system platform name (posix on Linux/macOS, nt on Windows).
  • {XEET_MATRIX_INDEX} — Zero-based index of the current matrix permutation.
  • {XEET_MATRIX_COUNT} — Total number of matrix permutations.
  • {XEET_MATRIX_PERMUTATION} — Current permutation mapping of variable names to values.

3. settings

Define configuration-wide defaults or reusable step templates:

settings:
  xeet:
    default_step_type: exec
  common_steps:
    ping_server:
      type: exec
      cmd: "curl -s {api_host}/health"
      allowed_rc: [0]
      expected_stdout: "OK\n"

4. tests

A list of test descriptors.

Test Model Fields

Field	Type	Description
name	string	Unique identifier for the test (^[a-zA-Z0-9_-]+$).
base	string	Name of another test to inherit from.
abstract	boolean	If true, the test cannot be run directly; only inherited by others.
short_desc	string	One-line description (displayed in listings).
long_desc	string	Detailed multi-line description (displayed in info).
groups	list[string]	Categorical tags used for filtering (-g, -G, -X).
variables	dict	Test-scoped variables overriding global variables.
matrix	dict	Test-scoped parameter matrix generating permutation variants.
platforms	list[string]	List of supported platforms (posix, nt). If set, test only runs on matching OS.
resources	list[resource_req]	Shared resources required by this test before execution.
pre_run	list[step]	Setup steps executed before the main phase.
run	list[step]	Main test steps.
post_run	list[step]	Tear-down steps executed after the main phase.
skip	boolean	If true, marks the test to be skipped.
skip_reason	string	Reason displayed when skipped.
expected_failure	boolean	Inverts result (passes if execution fails, fails if it passes).
inherit_matrix	boolean	true
pre_run_inheritance	prepend | append | replace	How inherited pre_run steps are combined (default: replace).
run_inheritance	prepend | append | replace	How inherited run steps are combined (default: replace).
post_run_inheritance	prepend | append | replace	How inherited post_run steps are combined (default: replace).

exec Step Fields

Field	Type	Default	Description
cmd	string	(required)	Shell or process command to execute.
base	string	""	JSONPath reference to a base step (e.g. settings.common_steps.ping_server).
cwd	string	null	Working directory for the process.
env	dict	{}	Environment variables to inject into the process.
env_file	string	null	Path to a JSON file containing environment variables.
use_os_env	boolean	false	Pass through the host operating system's environment.
use_shell	boolean	false	Execute command via shell (alias: shell).
shell_path	string	null	Custom shell executable path.
timeout	float	null	Maximum execution time in seconds before terminating.
allowed_rc	list[int] | "*"	[0]	Acceptable return codes (use "*" to accept any exit code).
output_behavior	unify | split	unify	Combine stdout/stderr into one stream, or split them.
stdout_file	string	"stdout"	Output capture filename for stdout.
stderr_file	string	"stderr"	Output capture filename for stderr (when split).
expected_stdout	string	null	Literal string to match against stdout.
expected_stdout_file	string	null	File path containing expected stdout.
expected_stderr	string	null	Literal string to match against stderr.
expected_stderr_file	string	null	File path containing expected stderr.
output_filters	list[filter]	[]	Text scrubbers applied to output before comparison.

5. resources

Define pools of shared resources that tests can acquire:

resources:
  # Named resources (accessible by specific name or FIFO)
  app_ports:
    - name: primary
      value: 8080
    - name: secondary
      value: 8081

  # Anonymous pool of interchangeable resources
  db_connections:
    - value: "postgres://localhost:5432/test1"
    - value: "postgres://localhost:5432/test2"

6. matrix

Define global matrix variables to execute tests across parameter combinations. Each variable defines a list of values, and xeet runs iterations across the full cartesian product of permutations:

matrix:
  browser: ["chrome", "firefox"]
  env_tier: ["staging", "prod"]

Matrix variables are available in test scopes like regular variables ({browser}, {env_tier}).

---

Inheritance Guide

xeet provides two distinct inheritance mechanisms: Test-level inheritance (for reusing setup phases, steps, and variables across tests) and Step-level inheritance (for sharing step definitions from a library or between tests).

1. Test Inheritance

A test can inherit from another test by specifying base: <parent_test_name>.

• Variables: Inherited by default (inherit_variables: true). A child test's variables override the parent's.
• Matrix: Inherited by default (inherit_matrix: true). A child test's matrix merges with the parent's, with child entries overriding parent entries.
• Platforms: Inherited as a whole list if unset on the child. A child test can override its base's platforms list, or explicitly set platforms: [] to clear an inherited restriction.
• Resources: Inherited as a whole list if unset on the child. A child test can override its base's resource requirements, or explicitly set resources: [] to clear an inherited restriction.
• Phase Steps (pre_run, run, post_run): Combined according to the phase inheritance policy:
  • replace (default): The child phase steps replace the parent's phase steps.
  • append: Parent steps execute first, followed by child steps.
  • prepend: Child steps execute first, followed by parent steps.
• Abstract Tests: Marked with abstract: true. Abstract tests cannot be run directly; they serve as templates for child tests and appear in xeet list -a.
• Non-Inherited Attributes: short_desc, long_desc, skip, and expected_failure are strictly test-specific and are deliberately never inherited.

Example: Abstract Base Test with Overrides and Appending

tests:
  - name: base_api_test
    abstract: true
    variables:
      endpoint: "/health"
    pre_run:
      - cmd: "curl -s -X POST http://localhost:8080/setup"
    run:
      - cmd: "curl -s http://localhost:8080{endpoint}"
        expected_stdout: "{\"status\": \"ok\"}\n"
    post_run:
      - cmd: "curl -s -X POST http://localhost:8080/teardown"

  # Inherits setup, teardown, and overrides the endpoint variable
  - name: test_user_profile
    base: base_api_test
    short_desc: Verify user profile endpoint
    groups: [api, smoke]
    variables:
      endpoint: "/api/v1/profile"

  # Inherits base and appends an additional verification step
  - name: test_with_metrics
    base: base_api_test
    short_desc: Verify endpoint and check metrics
    run_inheritance: append
    run:
      - cmd: "curl -s http://localhost:8080/metrics"
        expected_stdout: "metrics_collected: true\n"

---

2. Step Inheritance

Individual steps can inherit from reusable step templates defined in settings.common_steps or from steps in other tests using JSONPath via base: <jsonpath>.

The derived step automatically inherits all attributes from the base step (command, working directory, environment, timeouts, expected outputs) while overriding any explicitly specified fields.

Example: Reusing Common Steps with Overrides

settings:
  xeet:
    default_step_type: exec
  common_steps:
    # Base step template in the settings library
    run_python_script:
      type: exec
      cwd: "{XEET_ROOT}/scripts"
      env:
        PYTHONUNBUFFERED: "1"
      allowed_rc: [0]

tests:
  - name: data_processing
    run:
      # Inherits cwd, env, allowed_rc; specifies only cmd and expected output
      - base: "settings.common_steps.run_python_script"
        cmd: "python process.py --input data.csv"
        expected_stdout: "Processing complete.\n"

      # Re-uses the same template with custom environment override and timeout
      - base: "settings.common_steps.run_python_script"
        cmd: "python analyze.py"
        timeout: 10
        env:
          PYTHONUNBUFFERED: "1"
          ANALYSIS_MODE: "deep"

---

Output Filtering

For hermetic testing, output_filters allow scrubbing machine-specific details (such as absolute paths or timestamps) before diffing against expected baselines:

run:
  - cmd: "my_tool --report"
    expected_stdout_file: "baselines/report.txt"
    output_filters:
      - from_str: "{XEET_ROOT}"
        to_str: "__ROOT__"
      - from_str: "[0-9]+\\.[0-9]{3}s"
        to_str: "X.XXXs"
        regex: true

---

Platform-Specific Testing

xeet supports cross-platform test suites by allowing tests to restrict execution to specific operating systems (posix for Linux/macOS, nt for Windows) and by providing the {XEET_PLATFORM} auto-variable for conditional file inclusion.

1. Declaring Supported Platforms

Use the platforms field to restrict a test or an abstract base test to matching operating systems. Tests that do not match the host platform are automatically reported as Skipped:

tests:
  - name: test_posix_permissions
    platforms: [posix]
    run:
      - cmd: "ls -la /tmp"

  - name: test_windows_registry
    platforms: [nt]
    run:
      - cmd: "powershell -Command Get-ItemProperty 'HKCU:\\Software'"

2. Platform-Conditional Configuration Includes

Use {XEET_PLATFORM} in include directives to dynamically load OS-specific step libraries and variable definitions:

# Automatically loads 'xeet_posix.yaml' on Linux/macOS or 'xeet_nt.yaml' on Windows
include:
  - "common_steps.yaml"
  - "xeet_{XEET_PLATFORM}.yaml"

tests:
  - name: run_service
    base: platform_service_step

---

Resource Pools & Concurrency Control

When running tests in parallel (-j), tests may contend for shared physical or virtual resources (such as server ports, test accounts, database instances, or hardware devices). xeet provides resource pools to synchronize access and prevent conflicts without manually orchestrating execution order.

1. Declaring Resource Pools

Define one or more resource pools under the top-level resources section. Resources can be anonymous (defined by value) or named:

resources:
  # Named resources (accessible by specific name or FIFO)
  app_ports:
    - name: primary
      value: 8080
    - name: secondary
      value: 8081

  # Anonymous pool of interchangeable resources
  db_connections:
    - value: "postgres://localhost:5432/db1"
    - value: "postgres://localhost:5432/db2"

2. Requesting Resources in Tests

Tests declare resource requirements under the resources field. A test will only be scheduled when all requested resources are available:

• By Count: Request a number of interchangeable resources from a pool:

  tests:
    - name: test_db_migration
      resources:
        - pool: db_connections
          count: 1
          as_var: db_url
      run:
        - cmd: "migrate --database {db_url}"
  

• By Specific Names: Request specific named resources from a pool:

  tests:
    - name: test_primary_endpoint
      resources:
        - pool: app_ports
          names: [primary]
          as_var: port
      run:
        - cmd: "curl http://localhost:{port.primary}/status"
  

3. How Synchronization Works

• Automatic Deferral: If a test's required resources are currently in use by another thread, the runner automatically defers that test and executes other runnable tests, retrying the deferred test as soon as resources are released.
• Resource Variables (as_var): Acquired resources are injected as scoped test variables ({as_var_name}) accessible in step commands, environment variables, and working directories.
• Automatic Cleanup: Resources are automatically freed and their variables cleanly cleared when the test finishes (or fails), making them available for subsequent tests or iterations.

---

Parameter Matrix

The top-level matrix section defines variables with multiple candidate values. xeet computes the cartesian product of all matrix variables and executes the test suite across every permutation:

matrix:
  os_arch: ["x86_64", "arm64"]
  build_type: ["debug", "release"]

tests:
  - name: compile_and_test
    run:
      - cmd: "build --arch {os_arch} --mode {build_type}"

Permutation Selection & Filtering

Filter specific permutations directly on the command line:

# Run only permutation 0
xeet run -p 0

# Run permutations 0, 1, and 2
xeet run -p 0,1,2

# Exclude permutation 3
xeet run -P 3

Test-Level Matrix

Individual tests can also define their own matrix. When a test defines a matrix, it becomes an abstract template, and xeet automatically generates permutation variants using : notation (e.g. my_test:0, my_test:1):

tests:
  - name: db_stress_test
    matrix:
      connections: [10, 50, 100]
    run:
      - cmd: "stress_tool --conns {connections}"

Specific permutations can be queried or executed directly:

# Inspect a specific permutation
xeet info -t db_stress_test:0

# Run a specific permutation directly
xeet run -t db_stress_test:0

---

Command-Line Usage

Running Tests

# Run all tests in the default config (xeet.yaml)
xeet run

# Run with a specific configuration file
xeet run -c path/to/config.yaml

# Run specific tests by exact name or fuzzy pattern
xeet run -t test_name
xeet run -z smoke

# Filter by groups
xeet run -g integration              # Include any test in group 'integration'
xeet run -G api -G v2                # Require tests to belong to both 'api' AND 'v2'
xeet run -X slow                     # Exclude tests in group 'slow'

# Repeat execution (multi-iteration testing)
xeet run -r 5

# Parallel execution across worker threads
xeet run -j                          # Auto-detects half of CPU cores
xeet run -j 4                        # Run with 4 concurrent worker threads

# Test order randomization
xeet run --randomize                 # Randomize test execution order per iteration

# Matrix permutation selection
xeet run -p 0,1                      # Run only specific matrix permutations
xeet run -P 2                        # Exclude specific matrix permutations

# Control output verbosity
xeet run --verbose                   # Detailed per-test output, timings, criteria
xeet run --concise                   # Compact single-line output
xeet run --quiet                     # Minimal output
xeet run --debug                     # Live real-time process tailing and event logs

# Custom output directory for test artifacts
xeet run -O /path/to/artifacts

Inspecting Suites & Tests

# List all tests and one-line descriptions
xeet list

# List all tests including abstract base tests
xeet list -a

# List all available test groups
xeet groups

# Show detailed information about a test definition
xeet info -t my_test

# Show test information with resolved variables and full step details
xeet info -t my_test -x -f

Exporting & Schema Generation

# Dump the resolved JSON schema for IDE validation
xeet dump schema -t config
xeet dump schema -t test
xeet dump schema -t unified

# Dump a specific test descriptor in YAML
xeet dump test -t my_test

# Query and dump configuration sections using JSONPath
xeet dump config -p "settings.common_steps"

---

Exit Codes

xeet returns composite exit codes to facilitate CI/CD integration:

• 0 — All selected tests ran and passed.
• 1 — One or more tests failed.
• 2 — One or more tests could not be run (e.g. pre_run error or invalid command).
• 3 — Both test failures and unrunnable tests occurred (1 + 2).

---

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). See the LICENSE file for details.