test-exe-matrix 0.1.5

Toy project to test executables defined in a yaml file

A wrapper around py.test to run UNIX commands and check some of their behaviour.

Contents

• Installation and running

  • Main requirements

  • From PyPA (pip install)

    • Install

    • Run

  • From source with poetry

    • Install

    • Run

  • Command line options

• Parametrizing tests

  • Minimum viable test

  • Optional values

• Example

• Dev: Build the package

  • Other packaging

• Licence

• Thank you

Installation and running

Main requirements

• UNIX like

• Python 3.6, 3.7 or 3.8

From PyPA (pip install)

Install

The most recent packaged version can be installed with pip with options relevant to your usecase [1].

[1]

For instance pip{3} install --user test_exe_matrix, or install in a virtualenv.

pip install test_exe_matrix

Run

test_exe_matrix [arguments...]

From source with poetry

Install

Note on poetry: you can install it from pip (pip install --user poetry) [2] or follow the instructions on the official Web site.

[2]

They say:: Be aware that it will also install Poetry’s dependencies which might cause conflicts with other packages.

git clone https://framagit.org/feth/test_exe_matrix.git
cd test_exe_matrix
poetry install

Run

poetry run test_exe_matrix [arguments...]

Command line options

Typical use:

[poetry run] test_exe_matrix

Synopsys:

usage: test_exe_matrix [-h] [-V] [--collect-only] [-v] [--markers]
                       [-m PYTEST_ARGS]
                       [testsuite [testsuite ...]]

Test command lines listed in yaml files.

positional arguments:
  testsuite       testsuite yaml file -see example
                  test_exe_matrix/matrix.yaml.
                  Argument can be specified multiple times

optional arguments:
  -h, --help      show this help message and exit
  -V, --version   show program's version number and exit
  --collect-only  Only collect tests, without running them.
  -v              Increase verbosity.
  --markers       Lists registered test markers.
  -m PYTEST_ARGS  Only run tests that match selection expression, ie. '-m "not
                  internet"'

test_exe_matrix is mainly a Py.test wrapper

-v adds very nice colors, thanks to py.test.

Parametrizing tests

Put your test suites in a yaml, like matrix.yaml (provided), or in several. Every yaml file will have a separate global config (markers declaration and general timeout).

Minimum viable test

- exe: /path/to/exe

The test will succeed if default values are met: see below (optional values).

Optional values

timeout

Allowed running time in seconds. Program will be killed after this delay. Default to config/timeout or 1.

retcode

Expected return code.

stdout

Expected standard output (full match)

stderr

Expected standard error (full match)

partstdout

Substring expected in standard output.

partstderr

Substring expected in standard error.

notinstdout

Substring is not expected in standard output.

notinstderr

Substring is not expected in standard error.

args

List of arguments to your program.

name

Description of the test being made.

expect_too_long

Boolean: whether the program is expected to run over the time limit.

markers

List of markers for the test. You can use standard py.test markers (especially ‘xfail’: indicating ‘expected failure’) or custom markers, that _must_ be declared in the config section.

Example

A complete (hopefully) example file is provided as “matrix.yaml”.

Given the examples, you may run

test_exe_matrix matrix.yaml --collect-only

to list the tests, and

test_exe_matrix matrix.yaml -v -m "not internet and not slow"

to deselect all tests requiring Internet and slow tests.

Dev: Build the package

The project uses poetry. Do

poetry build

And a pip installable wheel package will appear under dist/. For instance, you can install with:

pip3 install dist/test_exe_matrix-0.0.18-py3-none-any.whl

Other packaging

Debian/Fedora/Conda packages would be a good thing, but I don’t know where to start. If I have to drop Poetry, let it be. Help welcome.

Licence

Dual licensed (at your option) to the public under CeCILL 2.1 and GPL 2, see files LICENSE_CeCILL_2.1.txt and LICENSE_GPL_2.txt.

Those are compatible copyleft licenses; CeCILL 2.1 is better suited for France at least.

Other needs: contact me.

Thank you

Stéphane Bortzmeyer, who once asked if a simple tool testing the result of commands existed, and then reported bugs/wrote helpful tickets on framagit/gitlab to guide the development.