Skip to main content

Toy project to test executables defined in a yaml file

Project description

A wrapper around py.test to run UNIX commands and check some of their behaviour.

made-with-python master-pipeline license-GPL-2 license-CeCILL-2.1 project-url repository-url

Installation and running

Main requirements

  • UNIX like

  • Python 3.6, 3.7 or 3.8

From PyPA (pip install)


The most recent packaged version can be installed with pip with options relevant to your usecase [1].

pip install test_exe_matrix


test_exe_matrix [arguments...]

From source with poetry


Note on poetry: you can install it from pip (pip install --user poetry) [2] or follow the instructions on the official Web site.

git clone
cd test_exe_matrix
poetry install


poetry run test_exe_matrix [arguments...]

Command line options

Typical use:

[poetry run] test_exe_matrix


usage: test_exe_matrix [-h] [-V] [--collect-only] [-v] [--markers]
                       [-m PYTEST_ARGS]
                       [testsuite [testsuite ...]]

Test command lines listed in yaml files.

positional arguments:
  testsuite       testsuite yaml file -see example
                  Argument can be specified multiple times

optional arguments:
  -h, --help      show this help message and exit
  -V, --version   show program's version number and exit
  --collect-only  Only collect tests, without running them.
  -v              Increase verbosity.
  --markers       Lists registered test markers.
  -m PYTEST_ARGS  Only run tests that match selection expression, ie. '-m "not

test_exe_matrix is mainly a Py.test wrapper

-v adds very nice colors, thanks to py.test.

Parametrizing tests

Put your test suites in a yaml, like matrix.yaml (provided), or in several. Every yaml file will have a separate global config (markers declaration and general timeout).

Minimum viable test

- exe: /path/to/exe

The test will succeed if default values are met: see below (optional values).

Optional values


Allowed running time in seconds. Program will be killed after this delay. Default to config/timeout or 1.


Expected return code.


Expected standard output (full match)


Expected standard error (full match)


Substring expected in standard output.


Substring expected in standard error.


Substring is not expected in standard output.


Substring is not expected in standard error.


List of arguments to your program.


Description of the test being made.


Boolean: whether the program is expected to run over the time limit.


List of markers for the test. You can use standard py.test markers (especially ‘xfail’: indicating ‘expected failure’) or custom markers, that _must_ be declared in the config section.


A complete (hopefully) example file is provided as “matrix.yaml”.

Given the examples, you may run

test_exe_matrix matrix.yaml --collect-only

to list the tests, and

test_exe_matrix matrix.yaml -v -m "not internet and not slow"

to deselect all tests requiring Internet and slow tests.

screenshot of the above command and output.

Dev: Build the package

The project uses poetry. Do

poetry build

And a pip installable wheel package will appear under dist/. For instance, you can install with:

pip3 install dist/test_exe_matrix-0.0.18-py3-none-any.whl

Other packaging

Debian/Fedora/Conda packages would be a good thing, but I don’t know where to start. If I have to drop Poetry, let it be. Help welcome.


Dual licensed (at your option) to the public under CeCILL 2.1 and GPL 2, see files LICENSE_CeCILL_2.1.txt and LICENSE_GPL_2.txt.

Those are compatible copyleft licenses; CeCILL 2.1 is better suited for France at least.

Other needs: contact me.

Thank you

Stéphane Bortzmeyer, who once asked if a simple tool testing the result of commands existed, and then reported bugs/wrote helpful tickets on framagit/gitlab to guide the development.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

test_exe_matrix-0.1.5.tar.gz (24.3 kB view hashes)

Uploaded source

Built Distribution

test_exe_matrix-0.1.5-py3-none-any.whl (24.2 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page