Main requirements

• UNIX like

• Python 3.6, 3.7 or 3.8

From PyPA (pip install)

pip install test_exe_matrix

test_exe_matrix [arguments...]

From source with poetry

git clone https://framagit.org/feth/test_exe_matrix.git
cd test_exe_matrix
poetry install

poetry run test_exe_matrix [arguments...]

Command line options

Typical use:

[poetry run] test_exe_matrix

Synopsys:

usage: test_exe_matrix [-h] [-V] [--collect-only] [-v] [--markers]
                       [-m PYTEST_ARGS]
                       [testsuite [testsuite ...]]

Test command lines listed in yaml files.

positional arguments:
  testsuite       testsuite yaml file -see example
                  test_exe_matrix/matrix.yaml.
                  Argument can be specified multiple times

optional arguments:
  -h, --help      show this help message and exit
  -V, --version   show program's version number and exit
  --collect-only  Only collect tests, without running them.
  -v              Increase verbosity.
  --markers       Lists registered test markers.
  -m PYTEST_ARGS  Only run tests that match selection expression, ie. '-m "not
                  internet"'

test_exe_matrix is mainly a Py.test wrapper

-v adds very nice colors, thanks to py.test.

Minimum viable test

- exe: /path/to/exe

The test will succeed if default values are met: see below (optional values).

Optional values

timeout

Allowed running time in seconds. Program will be killed after this delay. Default to config/timeout or 1.

retcode

Expected return code.

stdout

Expected standard output (full match)

stderr

Expected standard error (full match)

partstdout

Substring expected in standard output.

partstderr

Substring expected in standard error.

notinstdout

Substring is not expected in standard output.

notinstderr

Substring is not expected in standard error.

args

List of arguments to your program.

name

Description of the test being made.

expect_too_long

Boolean: whether the program is expected to run over the time limit.

markers

List of markers for the test. You can use standard py.test markers (especially ‘xfail’: indicating ‘expected failure’) or custom markers, that _must_ be declared in the config section.

Other packaging

Debian/Fedora/Conda packages would be a good thing, but I don’t know where to start. If I have to drop Poetry, let it be. Help welcome.