Skip to main content

Toy project to test executables defined in a yaml file

Project description

A wrapper around py.test to run UNIX commands and check some of their behaviour.

made-with-python master-pipeline license-GPL-2 license-CeCILL-2.1 project-url repository-url

Installation and running

Main requirements

  • UNIX like

  • Python 3.6, 3.7 or 3.8

From PyPA (pip install)

Install

The most recent packaged version can be installed with pip with options relevant to your usecase.

pip install test_exe_matrix

Run

test_exe_matrix [arguments...]

From source with poetry

Install

Note on poetry: you cannot install it from pip, you need to follow the instructions <https://python-poetry.org/docs/> _ on the official Web site.

git clone https://framagit.org/feth/test_exe_matrix.git
cd test_exe_matrix
poetry install

Run

poetry run test_exe_matrix [arguments...]

Command line options

Typical use:

[poetry run] test_exe_matrix

Synopsys:

usage: test_exe_matrix [-h] [-V] [--collect-only] [-v] [--markers]
                       [-m PYTEST_ARGS]
                       [testsuite [testsuite ...]]

Test command lines listed in yaml files.

positional arguments:
  testsuite       testsuite yaml file -see example
                  test_exe_matrix/matrix.yaml.
                  Argument can be specified multiple times

optional arguments:
  -h, --help      show this help message and exit
  -V, --version   show program's version number and exit
  --collect-only  Only collect tests, without running them.
  -v              Increase verbosity.
  --markers       Lists registered test markers.
  -m PYTEST_ARGS  Only run tests that match selection expression, ie. '-m "not
                  internet"'

test_exe_matrix is mainly a Py.test wrapper

-v adds very nice colors, thanks to py.test.

Parametrizing tests

Put your test suites in a yaml, like matrix.yaml (provided), or in several. Every yaml file will have a separate global config (markers declaration and general timeout).

Minimum viable test

- exe: /path/to/exe

The test will succeed if default values are met: see below (optional values).

Optional values

timeout

Allowed running time in seconds. Program will be killed after this delay. Default to config/timeout or 1.

retcode

Expected return code.

stdout

Expected standard output (full match)

stderr

Expected standard error (full match)

partstdout

Substring expected in standard output.

partstderr

Substring expected in standard error.

args

List of arguments to your program.

name

Description of the test being made.

expect_too_long

Boolean: whether the program is expected to run over the time limit.

markers

List of markers for the test. You can use standard py.test markers (especially ‘xfail’: indicating ‘expected failure’) or custom markers, that _must_ be declared in the config section.

Example

A complete (hopefully) example file is provided as “matrix.yaml”.

Given the examples, you may run

test_exe_matrix matrix.yaml --collect-only

to list the tests, and

test_exe_matrix matrix.yaml -v -m "not internet and not slow"

to deselect all tests requiring Internet and slow tests.

screenshot of the above command and output.

Dev: Build the package

The project uses poetry. Do

poetry build

And a pip installable wheel package will appear under dist/. For instance, you can install with:

pip3 install dist/test_exe_matrix-0.0.18-py3-none-any.whl

Licence

CeCILL 2.1 and GPL 2.1, see files LICENSE_CeCILL_2.1.txt and LICENSE_GPL_2.1.txt.

Thank you

Stéphane Bortzmeyer, who once asked if a simple tool testing the result of commands existed, and then reported bugs/wrote helpful tickets on framagit/gitlab to guide the development.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

test_exe_matrix-0.1.3.tar.gz (22.3 kB view hashes)

Uploaded Source

Built Distribution

test_exe_matrix-0.1.3-py3-none-any.whl (23.7 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page