qutest 8.0.0

QUTest Python scripting support

The qutest.py package is a test-script runner for the QUTest testing system.

General Requirements

In order to run tests in the QUTest environment, you need the following three components:

1. The test fixture in C or C++ running on a remote target (or the host computer)
2. The QSPY host application running and connected to the target
3. The qutest.py script runner and some test scripts.

NOTE: The qutest.py script runner requires standard Python 3, which is included in the QTools distribution for Windows and is typically included with other host operating systems, such as Linux and macOS.

Usage without Installation

The qutest.py script runner can be used standalone, without installation in your Python system (see Examples below).

REMARK: The latest qutest.py script is included in the QTools collection. Also, the QTools collection for Windows already includes Python 3, so you don't need to install anything extra.

Installation with pip

You can install the qutest package with the standard pip package manager.

NOTE: The qutest package is no longer available in the PyPi package index due to the overcomplicated "two-factor authentication".

Instead, you can direct pip to install directly from the qutest directory (e.g., /qp/qtools/qutest):

pip install /qp/qtools/qutest/qutest.tar.gz

Alternatively, you can direct pip to install from Quantum Leaps GitHub:

pip install https://github.com/QuantumLeaps/qtools/releases/latest/download/qutest.tar.gz

Using qutest.py

If you are using qutest.py as a standalone Python script, you invoke it as follows:

python3 <path-to-qutest-script>/qutest.py <command-line-options>

Alternatively, if you've installed qutest.py with pip, you invoke it as follows:

qutest <command-line-options>

Command-line Options

The Python test scripts are executed by the QUTest test script runner qutest.py (typically located in qtools/qutest/ folder), with the following usage:

ATTENTION The qutest.py script runner command-line options have been expanded and changed at version 7.2.0. Unfortunately, it was not possible to preserve the backwards compatibility with the earlier versions.

usage: python qutest.py [-h] [-v] [-e [EXE]] [-q [QSPY]] [-l [LOG]] [-o [OPT]] [scripts ...]

QUTest test script runner

positional arguments:
  scripts               List (comma-separated) of test scripts to run

options:
  -h, --help            show this help message and exit
  -v, --version         Display QUTest version
  -e [EXE], --exe [EXE]
                        Optional host executable or debug/DEBUG
  -q [QSPY], --qspy [QSPY]
                        optional qspy host, [:ud_port][:tcp_port]
  -l [LOG], --log [LOG]
                        Optional log directory (might not exist yet)
  -o [OPT], --opt [OPT]
                        xcob: x:exit-on-fail, c:qspy-clear, o:qspy-save-txt, b:qspy-save-bin

More info: https://www.state-machine.com/qtools/qutest.html

• -x - optional flag that causes qutest to exit on first test failure.

• test_scripts - optional specification of the Python test scripts to run. If not specified, qutest will try to run all *.py files in the current directory as test scripts

• host_exe | DEBUG - optional specification of the test-fixture compiled for the host (host executable) for testing on the host computer. The special value DEBUG means that qutest will run in the "debug mode", in which it will NOT launch the host executables and it will wait for the Target reset and other responses from the Target. If host_exe is not specified, an embedded target is assumed (which is loaded with the test fixture already).

• qspy_host[:udp_port] - optional host-name/IP-address:port for the host running the QSPY host utility. If not specified, the default is 'localhost:7701'.

• tcp_port - optional the QSpy TCP port number for connecting host executables. If not specified, the default is '6601'.

NOTE: For reliable operation it is recommended to apply the short options without a space between the option and the parameter (e.g., 1-q192.168.1.100, -ocx1).

Examples (for Windows):

[1] python3 %QTOOLS%\qutest\qutest.py
[2] python3 %QTOOLS%\qutest\qutest.py -- test_mpu.py
[3] python3 %QTOOLS%\qutest\qutest.py -ebuild/test_dpp.exe
[4] python3 %QTOOLS%\qutest\qutest.py -ebuild/test_dpp.exe -q192.168.1.100 -l../log -oco
[5] qutest -qlocalhost:7702 -oxc --  test_qk.py,test_mpu.py
[6] python3 %QTOOLS%\qutest\qutest.py -eDEBUG -- test_mpu.py

[1] runs all test scripts (*.py) in the current directory on a remote target connected to QSPU host utility.

[2] runs the test script test_mpu.py in the current directory on a remote target connected to QSPU host utility.

[3] runs all test scripts (*.py) in the current directory and uses the host executable build/test_dpp.exe (test fixture).

[4] runs all test scripts (*.py) in the current directory, uses the host executable build/test_dpp.exe (test fixture), and connects to QSPY running on a machine with IP address 192.168.1.100. Also produces QUTest log (-l) in the directory ../log. Also clears the QUTest screen before the run (-oc) and causes QSPY to save the text output to a file (-oo)

[5] runs "qutest" (installed with pip) to execute the test scripts test_qk.py,`test_mpu.py`` in the current directory, and connects to QSPY at UDP-host:port localhost:7701.

[6] runs "qutest" in the DEBUG mode to execute the test script test_mpu.py in the current directory.

Examples (for Linux/macOS):

[1] python3 $(QTOOLS)/qutest/qutest.py
[2] python3 $(QTOOLS)/qutest/qutest.py -- test_mpu.py
[3] python3 $(QTOOLS)/qutest/qutest.py -ebuild/test_dpp
[4] python3 $(QTOOLS)/qutest/qutest.py -ebuild/test_dpp -q192.168.1.100 -l../log -oco
[5] qutest -qlocalhost:7702 -oxc --  test_qk.py,test_mpu.py
[6] python3 $(QTOOLS)/qutest/qutest.py -eDEBUG -- test_mpu.py

[1] runs all test scripts (*.py) in the current directory on a remote target connected to QSPU host utility.

[2] runs the test script test_mpu.py in the current directory on a remote target connected to QSPU host utility.

[3] runs all test scripts (*.py) in the current directory and uses the host executable build/test_dpp (test fixture).

[4] runs all test scripts (*.py) in the current directory, uses the host executable build/test_dpp (test fixture), and connects to QSPY running on a machine with IP address 192.168.1.100. Also produces QUTest log (-l) in the directory ../log. Also clears the QUTest screen before the run (-oc) and causes QSPY to save the text output to a file (-oo)

[5] runs "qutest" (installed with pip) to execute the test scripts test_qk.py,test_mpu.py in the current directory, and connects to QSPY at UDP-host:port localhost:7701.

[6] runs "qutest" in the DEBUG mode to execute the test script test_mpu.py in the current directory.

Generating Test Logs

As required for safety certification, the qutest.py test runner can generate permanent records of the runs by producing log files. This feature is enabled by the -l<log-dir> command-line option.

The various make-files supplied in QP/C and QP/C++ allow you to supply the command-line options for saving QUTest logs (by defining the LOG= symbol while invoking make), for example:

[1] make LOG=.
[2] make LOG=../log
[3] make LOG=c:/cert/logs

[1] generates QUTest log file in the current directory (.)

[2] generates QUTest log file in the ../log directory (relative to the current directory)

[3] generates QUTest log file in the absolute directory c:/cert/logs

The following following listing shows the generated log file:

Run ID    : 221221_161550
Target    : build/test_qutest.exe

===================================[group]====================================
test_assert.py

This test group contains tests that intenionally FAIL,
to exercise failure modes of the QUTest system.

[ 1]--------------------------------------------------------------------------
Expected assertion
                                                             [ PASS (  0.1s) ]
[ 2]--------------------------------------------------------------------------
Unexpected assertion (should FAIL!)
  @test_assert.py:22
exp: "0000000002 COMMAND CMD_A 0"
got: "0000000002 =ASSERT= Mod=test_qutest,Loc=100"
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ![ FAIL (  0.2s) ]
[ 3]--------------------------------------------------------------------------
Simple passing test
                                                             [ PASS (  0.1s) ]
[ 4]--------------------------------------------------------------------------
Wrong assertion expectation (should FAIL!)
  @test_assert.py:32
exp: "0000000002 =ASSERT= Mod=test_qutest,Loc=200"
got: "0000000002 =ASSERT= Mod=test_qutest,Loc=100"
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ![ FAIL (  1.1s) ]
[ 5]--------------------------------------------------------------------------
Simple passing test
                                                             [ PASS (  0.1s) ]

=================================[ SUMMARY ]==================================

Target ID : 221221_161031 (QP-Ver=720)
Log file  : ./qutest221221_161550.txt
Groups    : 1
Tests     : 5
Skipped   : 0
FAILED    : 2 [ 2 4 ]

==============================[ FAIL (  2.7s) ]===============================

More Information

More information about the QUTest unit testing harness is available online at:

• https://www.state-machine.com/qtools/qutest.html

More information about the QP/QSPY software tracing system is available online at:

• https://www.state-machine.com/qtools/qpspy.html