SymTest

中文 | English

Note: This project was formerly known as cli-test-framework. Since version 1.3.0 it is released as symtest-cli; the old package name is no longer updated. Install with:

pip install symtest-cli

Regression testing for numerical and scientific command-line workflows.

Run solvers, compare HDF5/CSV outputs with tolerances, diagnose numerical regressions, and rerun only what failed.

SymTest can also be used as a general-purpose CLI regression framework: multi-step commands, large configuration sets, parallel execution, and CI-ready reports.

The project grew out of finite-element solver development, where a single test may run several programs, produce HDF5 or CSV results, compare them with tolerances, and track execution time across revisions.

What it solves

SymTest keeps the execution workflow and its acceptance criteria in one JSON or YAML configuration:

• Execute workflows — single commands or fail-fast step sequences, with timeouts, retries, variables, tags, and expected-failure support.
• Verify results — return codes, output text and regular expressions, plus text, JSON, CSV, XML, HDF5, binary, and custom-script file comparisons.
• Manage large suites — split configurations with import, reuse templates with extends, filter by name or tag, and locate cases across files with symtest find.
• Iterate and integrate — parallel execution, --last-failed, step-level --resume, runtime history, structured reports, and JUnit XML output.

The emphasis is practical: features are added to solve test workflows that occur in real CLI and scientific-computing projects.

Installation

Python 3.9 or newer is required.

pip install symtest-cli

YAML support is optional:

pip install "symtest-cli[yaml]"
pip install "symtest-cli[all]"

The default installation includes HDF5 and numerical comparison support.

Quick start

Create test_cases.json:

{
  "test_cases": [
    {
      "name": "hello",
      "command": "echo",
      "args": ["Hello World"],
      "tags": ["smoke"],
      "expected": {
        "return_code": 0,
        "output_contains": ["Hello World"]
      }
    }
  ]
}

Run it:

symtest run test_cases.json

Validate a configuration without executing it:

symtest validate test_cases.json

Numerical golden-file testing

File comparisons can be part of a test's acceptance criteria. Comparator parameters such as tolerances, table selection, filters, and encodings are declared next to the command:

{
  "test_cases": [
    {
      "name": "FEA displacement check",
      "command": "my_solver",
      "args": ["--input", "case1.dat", "--output", "out.h5"],
      "expected": {
        "return_code": 0,
        "output_contains": ["simulation finished"],
        "compare_files": [
          {
            "actual": "out.h5",
            "baseline": "ref/golden.h5",
            "rtol": 1e-5,
            "atol": 1e-8,
            "tables": ["NASTRAN/RESULT/NODAL/DISPLACEMENT"]
          },
          {
            "actual": "summary.csv",
            "baseline": "ref/summary.csv",
            "rtol": 1e-6
          }
        ]
      }
    }
  ]
}

The comparator type is inferred from the extension when type is omitted. Built-in types are text, json, csv, xml, h5, binary, and script. Workspace comparators can be added without changing the framework.

To intentionally accept changed outputs, use --update-baseline. Because this can overwrite reference files, interactive runs require typing yes; non-interactive runs must explicitly add --yes:

symtest run test_cases.json --update-baseline
symtest run test_cases.json --update-baseline --yes   # automation / CI

Keep baselines under version control and review every update.

Multi-step and iterative workflows

A case may contain an ordered steps list. Execution stops at the first failed step. For long workflows, --resume reuses saved state and skips steps that already passed:

symtest run solver_tests.json
symtest run solver_tests.json --last-failed
symtest run solver_tests.json -t long_case --resume

--resume deliberately trusts that workspace artifacts have not changed between runs.

Large test suites and cross-file search

Large suites can be divided into sub-configurations:

{
  "test_cases": [
    {"import": "cases/text_tests.json", "tags": ["text"]},
    {"import": "cases/h5_tests.json", "tags": ["h5", "regression"]}
  ]
}

symtest find auto-expands all import references and searches across the unified case set (substring / fuzzy / regex modes), making it easy to locate cases and review scenario coverage in large projects:

symtest find main_config.json "login"

Parallel execution and resources

symtest run test_cases.json --parallel --workers 4
symtest run test_cases.json --parallel --execution-mode process

Thread mode supports CPU-token allocation, solver thread environment variables, and longest-processing-time-first scheduling using estimates or runtime history. On Windows, enabling --history-dir or declaring min_memory_mb additionally turns on opportunistic memory admission: each case's process-group commit-memory peak is measured via a Job Object and stored in history, and a case starts only when CPU tokens, commit margin (the case's own estimated demand plus a reserve), a physical-memory watermark, and (only under potential peak crowding) a minimum startup interval are all satisfied — queued instead of launching into a memory shortage. Active tasks hold no full peak reservations, so occasional overcommit is accepted (zero-OOM not guaranteed). Process mode provides execution isolation but does not yet use the resource scheduler. Linux memory scheduling, priority semantics, and broader resource scheduling remain active areas of development.

CI and reports

symtest run test_cases.json \
  --parallel --workers 4 \
  --junit-xml report.xml

The current suite contains 750 unit, integration, and end-to-end tests with 83% line coverage. CI exercises Windows and Linux across Python 3.9 through 3.13.

Python API

from symtest.runners import JSONRunner, ParallelJSONRunner

runner = ParallelJSONRunner(
    config_file="test_cases.json",
    max_workers=4,
    execution_mode="thread",
    history_dir="./hist",
    variables={"solver": "/opt/solver/bin/solver"},
)

success = runner.run_tests()
for detail in runner.results["details"]:
    print(detail["name"], detail["status"], detail.get("duration"))

Standalone file comparison

compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5
compare-files data1.csv data2.csv --csv-rtol 1e-4 --csv-data-filter ">1e-6"
compare-files data1.json data2.json --json-compare-mode key-based --json-key-field id

AI-assisted TDD, as a side benefit

The same configuration can serve as a machine-readable acceptance contract. Structured validation failures, comparison details, and targeted reruns work well in an AI-assisted TDD loop:

define acceptance criteria
    → run the relevant cases
    → inspect the structured failure
    → change the implementation
    → rerun with --last-failed
    → run the full regression suite

This is a useful consequence of explicit tests and structured results, not a requirement for using the framework.

Documentation

• User manual
• Design document
• Plugin examples
• Official AI Skill — install this skill into AI coding assistants so they can author test cases, acceptance criteria, and TDD workflows directly — the recommended entry point for new users

Development

Install the package with all optional and test dependencies:

pip install -e ".[dev]"
python -m pytest tests/unit tests/integration tests/e2e

Bug fixes, comparator plugins, documentation improvements, and reports from real-world test workflows are welcome.

Follow the development guidelines for documentation maintenance. After changing the English manual or official Python runner example, run python tools/sync_skill_assets.py to refresh their skill copies. Before delivery, run python tools/sync_skill_assets.py --check; CI also runs this check.

License

MIT