BRunhilda the test system

Brunhilda is the test runner with advanced reporting features.

Documentation

• How to run tests and create test reports.
• How to read test reports.

Contributing

Here is a short guide to the environment setup to ease you up contributing to the project. Start by installing and creating a virtual environment

pip install virtualenv
virtualenv venv

Now you should see venv folder in the project structure. Activate virtual environment

source venv/bin/activate    # Linux
venv\Scripts\activate.bat   # Windows

After that you should see (venv) as a prefix to your command line cursor. You have to repeat activation every time you close the terminal. To install the package in development mode call:

pip install -e .

If you want to use Brunhilda for user stories defined by doorstop artifacts then install BRunhilda by calling a command below. Please note that BRunhilda with doorstop feature requires Python >= 3.9.

pip install -e .[doorstop-user-stories]

Now you can use BRunhilda package directly from the command line and all changes to the source code are instantly applied.

Testing

The project includes both unit tests and integration tests.

Integration Tests

Integration tests verify report generation and prevent regressions. They support two execution modes depending on whether backward compatibility needs to be verified:

• Backward-Compatibility Testing (Two Datasets): Used for bug fixes where reports must be validated against datasets produced by older runner versions.

  1. Original Dataset (<runner_version>): Anonymized execution data captured from the specific runner version where the bug was reported.
  2. Current Dataset (head): Fresh execution data generated dynamically by running sample tests on the current version.

• Single-Dataset Testing: Used when testing general report features, layout, or formatting without needing legacy version comparisons. Only the current dataset is required, and the test fixture automatically skips the legacy execution pass if no legacy dataset folder is present.

Directory Structure

Integration tests are organized by bug tickets (test/integration/<bug_ticket_number>/):

test/integration/14492/
├── test_14492_report_generation.py
└── data/
    ├── 2.8.1/    # Original data from the bug ticket
    └── head/     # Data generated from current execution

Visual Studio Code tasks

To speed up the execution of useful operations for development, vscode tasks are implemented.

• static-type-check Static type checking using mypy
• test-with-coverage Running unit tests including measuring code coverage.
• integration-tests Running integration tests.

Tasks can be run within vscode Terminal -> Run Tasks...