CLI Test Framework

A lightweight automated testing framework for command-line applications. Define test cases in JSON/YAML, run all validations with a single command.

Particularly suited for scientific computing — deep HDF5 support with regex table matching, data filtering, and tolerance-based comparison, making simulation result verification effortless.

Why this exists

This project was created from a real need: regression testing finite-element software.

In solver-style projects, checking the exit code is not enough. A test often needs to run multiple commands, generate numerical result files, compare HDF5/CSV outputs with tolerances, and make sure old cases do not become slower over time.

CLI Test Framework was built for that workflow.

Highlights

• Golden File Assertion — compare_files embedded in test expected, compares output files against baselines with tolerance
• Parallel Execution — Multi-thread / multi-process, 3-5x speedup
• Resource-Aware Scheduling — Automatic CPU core management, prevents solver thread runaway
• Sequence Steps — Multi-step execution within a single test case, fail-fast
• Setup Module — Auto-configure environment variables before tests, auto-cleanup after
• File Comparison — Text / JSON / CSV / XML / HDF5 / Binary, with CLI and embedded assertion support
• Filtered Execution — Run specific test cases by name or tag
• JUnit XML Output — --junit-xml for GitLab CI / Jenkins / CircleCI test report panels

Quick Start

pip install cli-test-framework

30-Second Setup

1. Create test_cases.json:

{
    "test_cases": [
        {
            "name": "hello",
            "command": "echo",
            "args": ["Hello World"],
            "tags": ["smoke"],
            "expected": {
                "return_code": 0,
                "output_contains": ["Hello World"]
            }
        }
    ]
}

2. Run:

cli-test run test_cases.json

Golden File Comparison in Tests

Run a simulation, then compare its output file against a reference:

{
    "test_cases": [
        {
            "name": "FEA displacement check",
            "command": "my_solver",
            "args": ["--input", "case1.dat", "--output", "out.h5"],
            "expected": {
                "return_code": 0,
                "compare_files": [
                    {
                        "actual": "out.h5",
                        "baseline": "ref/golden.h5",
                        "rtol": 1e-5,
                        "atol": 1e-8,
                        "tables": ["NASTRAN/RESULT/NODAL/DISPLACEMENT"]
                    }
                ]
            }
        }
    ]
}

• actual – file produced by the command
• baseline – reference file to compare against
• type – comparator type (auto-detected from extension if omitted: .h5→h5, .json→json, .csv→csv, .xml→xml, .txt→text)
• all other keys are forwarded as comparator parameters (rtol, atol, tables, table_regex, data_filter, encoding, structure_only, delimiter, compare_mode, key_field, etc.)

Multiple files and mixed assertion types coexist naturally:

{
    "expected": {
        "return_code": 0,
        "output_contains": ["simulation finished"],
        "compare_files": [
            {"actual": "out.h5", "baseline": "ref/disp.h5", "rtol": 1e-5},
            {"actual": "report.csv", "baseline": "ref/expected.csv", "rtol": 1e-6}
        ]
    }
}

Parallel Execution

cli-test run test_cases.json --parallel --workers 4

Run Specific Cases

# By name
cli-test run test_cases.json -t test_1 -t test_2

# By tag
cli-test run test_cases.json --tag smoke

# Combine name and tag (AND logic)
cli-test run test_cases.json -t test_1 --tag smoke

JUnit XML (CI Integration)

cli-test run test_cases.json --junit-xml report.xml

In GitLab CI:

# .gitlab-ci.yml
artifacts:
  reports:
    junit: report.xml

Python API

from cli_test_framework.runners import JSONRunner, ParallelJSONRunner

# Sequential
runner = JSONRunner(config_file="test_cases.json")
success = runner.run_tests()

# Parallel
runner = ParallelJSONRunner(config_file="test_cases.json", max_workers=4, execution_mode="thread")
success = runner.run_tests()

File Comparison (Standalone CLI)

compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5

📖 Full Documentation: docs/user_manual.md

Changelog

0.9.0

• Tag filtering: New tags field in test case definitions for categorization. Use --tag CLI option or test_case_tag_filter Python API to batch-filter test cases. Acts as OR within tags, AND with name filtering.

0.8.0

• JUnit XML output: New --junit-xml <filepath> CLI option writes JUnit-format XML reports that integrate directly with GitLab CI, Jenkins, CircleCI, and other CI tools. Supports passed / failed / timeout / error status mapping. Zero extra dependencies — built with xml.etree.ElementTree.

0.7.0

• Unified logging system: All diagnostic output (executor, runner, scheduler, setup) now uses Python's standard logging module under the cli_test_framework namespace. Library users can suppress output entirely via logging.getLogger("cli_test_framework").setLevel(logging.WARNING). Removes the previous ad-hoc print() + _print_lock pattern — logging is inherently thread-safe.
• Default handler: A StreamHandler at INFO level is installed on first import, so CLI behavior is unchanged. Use --verbose / --debug to enable DEBUG-level output.
• Public API: get_logger(name) exposed via cli_test_framework.get_logger for consistent logger creation in extensions.

0.6.0

• Golden file assertion: compare_files is now a first-class assertion in test expected — compare output files against baselines directly in your test definitions, with full tolerance and parameter support. The file_comparator subsystem is now integrated into the assertion pipeline, completing the closed loop from command execution to result verification.

0.5.2

• Runtime history tracking (--history-dir): persist per-case execution time in .symtest, enable smart scheduling & regression detection
• Regression warning: alert when a case runs slower than historical average × threshold (--regression-threshold, default 1.5)
• Smart scheduling: parallel runner prioritizes historical avg_duration over config estimated_time for task ordering
• Per-case duration now shown in test result output

0.5.1

• Run specific test cases by name (-t / test_case_filter)

0.5.0

• Steps feature: sequential multi-command execution within a single test case, fail-fast

0.4.2

• Resource-aware scheduling: auto-detect CPU cores, semaphore-based core allocation
• Auto-inject OMP_NUM_THREADS / MKL_NUM_THREADS / NPROC to prevent solver thread runaway
• Per-test timeout support to prevent hanging

0.4.1

• Multi-thread / multi-process parallel execution, 3-5x speedup

Contributing

Before submitting a PR, please make sure all tests pass:

python tests\run_all.py

License

MIT