testsuit

Toolbox for preparing and analysing test data: load measurement files (CSV, TDMS, MDF, Dewesoft DXD, UDBF, HDF5, InfluxDB) into pandas, convert them to HDF5 or push them to a database, check results against YAML scenarios, and package test inputs into datapacks.

Installation

Python 3.10 or newer is required.

git clone https://github.com/laurentmldev/testsuit.git
cd testsuit
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .

Optional features are installed as extras:

For example pip install -e ".[plotly,jupyter]".

To use testsuit without cloning it, install straight from GitHub (add @<tag> to pin a version):

pip install "testsuit[jupyter] @ git+https://github.com/laurentmldev/testsuit.git"

Jupyter GUIs

Three notebook GUIs come with the package: Data-Plot (browse and plot parameters), H5-Conv (convert data files to HDF5) and M-Exploit (run a scenario and view its report). Copy their starter notebooks into a working folder and open Jupyter there:

pip install "testsuit[jupyter]"          # or: pip install -e ".[jupyter]" from a clone
testsuit-notebooks ~/my_analysis --open  # copies data_plot, h5_convert and mexploit .ipynb

The notebooks only call runGUI() from the installed package, so upgrading testsuit upgrades the GUIs without copying the notebooks again (--force refreshes them). The GUIs need the classic notebook UI (jupyter nbclassic, what --open starts), and a desktop session for their file dialogs (tkinter).

Plots

Figures (Data-Plot GUI, mexploit plot* checks, sequence timelines) are drawn by a plot backend:

Pick it per check with rendering_engine: plotly in the scenario YAML, per run with mxp --plot-backend plotly / exploit_runner --plot-backend plotly (or the PLOT_RENDERING_BACKEND environment variable), and in the Data-Plot GUI with its Backend menu. The figure format works the same way: figure_format, --figure-format, PLOT_FIGURE_FORMAT.

The exploit_runner HTML report stays a single file that works offline with plotly figures: they are stored as gzipped plotly JSON and drawn by plotly.js, which the report then embeds once (about 2 MB). Reports without plotly figures don't change. Another library can add its own backend with testsuit.datatools.plotBackends.register_plot_backend().

Repository layout

src/testsuit/           the installable package
  datatools/            load data files into pandas, convert to HDF5, push to a DB
    plotBackends/       matplotlib and plotly drawing of plotHelpers' figures
    DataFileMgrs/       one reader per file format (Csv, Tdms, Mdf, Dxd, Udbf, H5, InfluxDb)
    datapack/           datapack generation: dictionaries, includes, key replacement
  exploit/
    mexploit/           YAML scenario checks (corridors, sequences, computed params, plots) run through pytest
    runner/             runs mexploit over test sessions and builds HTML reports
  jupytertools/         ipywidgets GUIs for notebooks
  misc/                 logging, progress reporting, file helpers
  cli/                  command-line tools (see below)
examples/               sample configuration files, and acme_testbench: a library extending testsuit
tests/                  pytest suite (*_t.py)
  etc/                  input data, scenarios and configs used by the tests
  ref/                  reference outputs the tests compare against

Command-line tools

Installing the package puts these commands on your PATH. Each prints its full usage with --help, for example data2h5 --help. They can also be run as python -m testsuit.cli.<name>.

Using the library

from testsuit.datatools.datatoolbox import loadDataframeFromFile
from testsuit.exploit.mexploit.mexploit import mexploit

# run the checks of a scenario folder on a data folder, results written to out/
dfs = loadDataframeFromFile("tests/etc/data/csv/flags.csv", ".*")  # one DataFrame per parameter

rc = mexploit("tests/etc/mexploit/scenarii/mxp_OK_main_functions", "tests/etc/data", "out/", force=True)

The scenarios under tests/etc/mexploit/scenarii/ are working examples of every check type.

Running the tests

pip install -e ".[dev]"
pytest          # runs tests/*_t.py
ruff check .    # catches runtime errors such as undefined names

CI (.github/workflows/tests.yml) runs the same lint, the tests on Python 3.10 and 3.12, and a package build, on every pull request.

See CONTRIBUTING.md for how to add a file format, a mexploit check or a script, and how another library can register its own formats, criteria and report logo.

License

Apache License 2.0, see LICENSE.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR ITS COMPANY BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.