Pyst - Pytest System-Test Plugin

Note: This plugin is the state experimental / techdemo !

This plugin should ease system-testing with Pytest. To achieve this it provides a simple interface to manage processes in the operating system like starting them, running background processes and evaluating the output.

This is especially useful to prepare the system for a test (e.g. start background prcesses, databases, servers, ...) and then run the programm under test in a blackbox-manner.

Features should be (intended, partly unfinished):

• DONE: Starting, stopping and managing processes in the OS
• DONE: Pysys like interface to run systemtests
  • Especially startProcess and assertGrep
• DONE: Logs all outputs to files and can later on run assertions on that outputs
• Should work:
  • DONE: Start and stop processes with and without sudo
  • Prepare configuration files for executables
• DONE: Simple user fixtures
• Provides envioronment fixtures / group fixtures / environments (terminology depends on the framework you are used to).
• Multistage environments fixtures
  • Supports you to write your own fixures

So far, this will only un on unix-like systems.

Prepare

python3 -m venv env-plugin
source env-plugin/bin/activate
pip install pytest
pip install pytest-mock
pip install restapi_echo_server
or
pip install -r requirements.txt

Usage

Have a look at tests/test_use_cases.py it contains some exemplary use-cases. We use an echo-server and curl to be near to real use cases. More will be added soon.

Example

Some other use cases are here https://github.com/abelikt/pytest_system_test_plugin_test

import re
import pytest

def test_use_rolldice(process_factory):
    """This test installs a package via the package management, uses
    it and removes it again.
    """

    # make sure it is not installed
    call = process_factory(["whereis", "rolldice"])
    call.run()
    assert call.get_stdout() == "rolldice:"
    assert call.returncode == 0

    # Install package
    install = process_factory(["sudo", "apt-get", "install", "-y", "-q", "rolldice"])
    install.run()
    assert install.returncode == 0

    # Call and check output against a regex
    call = process_factory(["rolldice", "6"])
    call.run()
    print(call.get_stdout())
    print(call.get_stderr())

    # match a single digit
    assert re.match(r"^\d$", call.get_stdout()) is not None
    assert call.returncode == 0

    # deinstall it
    deinstall = process_factory(["sudo", "apt-get", "remove", "-y", "-q", "rolldice"])
    deinstall.run()
    assert install.returncode == 0

Test

pip install .; python -m pytest -s --log-cli-level=INFO
tox
tox -e {lint, linttest}

Measure test execution

pip install .; python -m pytest -s --log-cli-level=INFO --junitxml=junit.xml
xmllint junit.xml --format > junitf.xml

Links

This repo:

• https://gitlab.com/bitmuster/pytest_system_test_plugin
• https://github.com/bitmuster/pytest_system_test_plugin

Some exemplary tests:

• https://github.com/abelikt/pytest_system_test_plugin_test

Pytest test-framework:

• https://docs.pytest.org

PySys test-framework:

• https://pysys-test.github.io/pysys-test

Similar plugins:

• https://github.com/pysys-test/pysys-test
• https://github.com/CS-SI/pytest-executable