Toy project to test executables defined in a yaml file
Project description
A wrapper around py.test to run UNIX commands and check some of their behaviour.
Contents
Installation and running
Main requirements
- UNIX like
- Python 3.6, 3.7 or 3.8
From PyPA (pip install)
Install
The most recent packaged version can be installed with pip with options relevant to your usecase [1].
[1] | For instance pip{3} install --user test_exe_matrix, or install in a virtualenv. |
pip install test_exe_matrix
Run
test_exe_matrix [arguments...]
From source with poetry
Install
Note on poetry: you can install it from pip (pip install --user poetry) [2] or follow the instructions on the official Web site.
[2] | They say:: Be aware that it will also install Poetry’s dependencies which might cause conflicts with other packages. |
git clone https://framagit.org/feth/test_exe_matrix.git
cd test_exe_matrix
poetry install
Run
poetry run test_exe_matrix [arguments...]
Command line options
Typical use:
[poetry run] test_exe_matrix
Synopsys:
usage: test_exe_matrix [-h] [-V] [--collect-only] [-v] [--markers]
[-m PYTEST_ARGS]
[testsuite [testsuite ...]]
Test command lines listed in yaml files.
positional arguments:
testsuite testsuite yaml file -see example
test_exe_matrix/matrix.yaml.
Argument can be specified multiple times
optional arguments:
-h, --help show this help message and exit
-V, --version show program's version number and exit
--collect-only Only collect tests, without running them.
-v Increase verbosity.
--markers Lists registered test markers.
-m PYTEST_ARGS Only run tests that match selection expression, ie. '-m "not
internet"'
test_exe_matrix is mainly a Py.test wrapper
-v adds very nice colors, thanks to py.test.
Parametrizing tests
Put your test suites in a yaml, like matrix.yaml (provided), or in several. Every yaml file will have a separate global config (markers declaration and general timeout).
Minimum viable test
- exe: /path/to/exe
The test will succeed if default values are met: see below (optional values).
Optional values
- timeout
- Allowed running time in seconds. Program will be killed after this delay. Default to config/timeout or 1.
- retcode
- Expected return code.
- stdout
- Expected standard output (full match)
- stderr
- Expected standard error (full match)
- partstdout
- Substring expected in standard output.
- partstderr
- Substring expected in standard error.
- notinstdout
- Substring is not expected in standard output.
- notinstderr
- Substring is not expected in standard error.
- args
- List of arguments to your program.
- name
- Description of the test being made.
- expect_too_long
- Boolean: whether the program is expected to run over the time limit.
- markers
- List of markers for the test. You can use standard py.test markers (especially ‘xfail’: indicating ‘expected failure’) or custom markers, that _must_ be declared in the config section.
Example
A complete (hopefully) example file is provided as “matrix.yaml”.
Given the examples, you may run
test_exe_matrix matrix.yaml --collect-only
to list the tests, and
test_exe_matrix matrix.yaml -v -m "not internet and not slow"
to deselect all tests requiring Internet and slow tests.
Dev: Build the package
The project uses poetry. Do
poetry build
And a pip installable wheel package will appear under dist/. For instance, you can install with:
pip3 install dist/test_exe_matrix-0.0.18-py3-none-any.whl
Other packaging
Debian/Fedora/Conda packages would be a good thing, but I don’t know where to start. If I have to drop Poetry, let it be. Help welcome.
Licence
Dual licensed (at your option) to the public under CeCILL 2.1 and GPL 2, see files LICENSE_CeCILL_2.1.txt and LICENSE_GPL_2.txt.
Those are compatible copyleft licenses; CeCILL 2.1 is better suited for France at least.
Other needs: contact me.
Thank you
Stéphane Bortzmeyer, who once asked if a simple tool testing the result of commands existed, and then reported bugs/wrote helpful tickets on framagit/gitlab to guide the development.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size test_exe_matrix-0.1.5-py3-none-any.whl (24.2 kB) | File type Wheel | Python version py3 | Upload date | Hashes View |
Filename, size test_exe_matrix-0.1.5.tar.gz (24.3 kB) | File type Source | Python version None | Upload date | Hashes View |
Hashes for test_exe_matrix-0.1.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1f69d0b94c52fcc856c5f0ea96cc446c64697b5a492549b8bedfd72fd94c850 |
|
MD5 | fc96ec4187f057a22735da1aa754f73e |
|
BLAKE2-256 | 26cd124c97e831e0f9c47d6cb7877595c990eea966c791836c15d07710ba5be0 |