Skip to main content

Toy project to test executables defined in a yaml file

Project description

A wrapper around py.test to run UNIX commands and check some of their behaviour.

made-with-python master-pipeline license-GPL-2 license-CeCILL-2.1 project-url repository-url

Installation and running

Main requirements

  • UNIX like
  • Python 3.6, 3.7 or 3.8

From PyPA (pip install)

Install

The most recent packaged version can be installed with pip with options relevant to your usecase [1].

[1]For instance pip{3} install --user test_exe_matrix, or install in a virtualenv.
pip install test_exe_matrix

Run

test_exe_matrix [arguments...]

From source with poetry

Install

Note on poetry: you can install it from pip (pip install --user poetry) [2] or follow the instructions on the official Web site.

[2]They say:: Be aware that it will also install Poetry’s dependencies which might cause conflicts with other packages.
git clone https://framagit.org/feth/test_exe_matrix.git
cd test_exe_matrix
poetry install

Run

poetry run test_exe_matrix [arguments...]

Command line options

Typical use:

[poetry run] test_exe_matrix

Synopsys:

usage: test_exe_matrix [-h] [-V] [--collect-only] [-v] [--markers]
                       [-m PYTEST_ARGS]
                       [testsuite [testsuite ...]]

Test command lines listed in yaml files.

positional arguments:
  testsuite       testsuite yaml file -see example
                  test_exe_matrix/matrix.yaml.
                  Argument can be specified multiple times

optional arguments:
  -h, --help      show this help message and exit
  -V, --version   show program's version number and exit
  --collect-only  Only collect tests, without running them.
  -v              Increase verbosity.
  --markers       Lists registered test markers.
  -m PYTEST_ARGS  Only run tests that match selection expression, ie. '-m "not
                  internet"'

test_exe_matrix is mainly a Py.test wrapper

-v adds very nice colors, thanks to py.test.

Parametrizing tests

Put your test suites in a yaml, like matrix.yaml (provided), or in several. Every yaml file will have a separate global config (markers declaration and general timeout).

Minimum viable test

- exe: /path/to/exe

The test will succeed if default values are met: see below (optional values).

Optional values

timeout
Allowed running time in seconds. Program will be killed after this delay. Default to config/timeout or 1.
retcode
Expected return code.
stdout
Expected standard output (full match)
stderr
Expected standard error (full match)
partstdout
Substring expected in standard output.
partstderr
Substring expected in standard error.
notinstdout
Substring is not expected in standard output.
notinstderr
Substring is not expected in standard error.
args
List of arguments to your program.
name
Description of the test being made.
expect_too_long
Boolean: whether the program is expected to run over the time limit.
markers
List of markers for the test. You can use standard py.test markers (especially ‘xfail’: indicating ‘expected failure’) or custom markers, that _must_ be declared in the config section.

Example

A complete (hopefully) example file is provided as “matrix.yaml”.

Given the examples, you may run

test_exe_matrix matrix.yaml --collect-only

to list the tests, and

test_exe_matrix matrix.yaml -v -m "not internet and not slow"

to deselect all tests requiring Internet and slow tests.

screenshot of the above command and output.

Dev: Build the package

The project uses poetry. Do

poetry build

And a pip installable wheel package will appear under dist/. For instance, you can install with:

pip3 install dist/test_exe_matrix-0.0.18-py3-none-any.whl

Other packaging

Debian/Fedora/Conda packages would be a good thing, but I don’t know where to start. If I have to drop Poetry, let it be. Help welcome.

Licence

Dual licensed (at your option) to the public under CeCILL 2.1 and GPL 2, see files LICENSE_CeCILL_2.1.txt and LICENSE_GPL_2.txt.

Those are compatible copyleft licenses; CeCILL 2.1 is better suited for France at least.

Other needs: contact me.

Thank you

Stéphane Bortzmeyer, who once asked if a simple tool testing the result of commands existed, and then reported bugs/wrote helpful tickets on framagit/gitlab to guide the development.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for test-exe-matrix, version 0.1.5
Filename, size File type Python version Upload date Hashes
Filename, size test_exe_matrix-0.1.5-py3-none-any.whl (24.2 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size test_exe_matrix-0.1.5.tar.gz (24.3 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page