cli-test-framework 0.5.4

A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.

CLI Test Framework

A lightweight automated testing framework for command-line applications. Define test cases in JSON/YAML, run all validations with a single command.

Particularly suited for scientific computing — deep HDF5 support with regex table matching, data filtering, and tolerance-based comparison, making simulation result verification effortless.

Why this exists

This project was created from a real need: regression testing finite-element software.

In solver-style projects, checking the exit code is not enough. A test often needs to run multiple commands, generate numerical result files, compare HDF5/CSV outputs with tolerances, and make sure old cases do not become slower over time.

CLI Test Framework was built for that workflow.

Highlights

• Parallel Execution — Multi-thread / multi-process, 3-5x speedup
• Resource-Aware Scheduling — Automatic CPU core management, prevents solver thread runaway
• Sequence Steps — Multi-step execution within a single test case, fail-fast
• Setup Module — Auto-configure environment variables before tests, auto-cleanup after
• File Comparison — Text / JSON / HDF5 / Binary, ready to use from command line
• Filtered Execution — Run specific test cases by name

Quick Start

pip install cli-test-framework

30-Second Setup

1. Create test_cases.json:

{
    "test_cases": [
        {
            "name": "hello",
            "command": "echo",
            "args": ["Hello World"],
            "expected": {
                "return_code": 0,
                "output_contains": ["Hello World"]
            }
        }
    ]
}

2. Run:

cli-test run test_cases.json

Parallel Execution

cli-test run test_cases.json --parallel --workers 4

Python API

from cli_test_framework.runners import JSONRunner, ParallelJSONRunner

# Sequential
runner = JSONRunner(config_file="test_cases.json")
success = runner.run_tests()

# Parallel
runner = ParallelJSONRunner(config_file="test_cases.json", max_workers=4, execution_mode="thread")
success = runner.run_tests()

File Comparison

compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5

📖 Full Documentation: docs/user_manual.md

Changelog

0.5.2

• Runtime history tracking (--history-dir): persist per-case execution time in .symtest, enable smart scheduling & regression detection
• Regression warning: alert when a case runs slower than historical average × threshold (--regression-threshold, default 1.5)
• Smart scheduling: parallel runner prioritizes historical avg_duration over config estimated_time for task ordering
• Per-case duration now shown in test result output

0.5.1

• Run specific test cases by name (-t / test_case_filter)

0.5.0

• Steps feature: sequential multi-command execution within a single test case, fail-fast

0.4.2

• Resource-aware scheduling: auto-detect CPU cores, semaphore-based core allocation
• Auto-inject OMP_NUM_THREADS / MKL_NUM_THREADS / NPROC to prevent solver thread runaway
• Per-test timeout support to prevent hanging

0.4.1

• Multi-thread / multi-process parallel execution, 3-5x speedup

Contributing

Before submitting a PR, please make sure all tests pass:

python tests\run_all.py

License

MIT