qecore 4.12

DesktopQE Tool for unified test execution

qecore - Library of tools

Qecore is a Library of Tools designed to complement our automation stack with all that is required.

Usage

The qecore-headless script.

• The name headless in the name is historical, it is a session configuration script that runs our automation stack.
• This script provides a way to start a new session in ssh connected machine.
• It serves only for remote usage, using it locally will not end well. It is designed for fast GDM start and stop with user autologin.
• This script will start GDM and make a few changes to the system to enable start of an automation suite in wanted configuration, for example:
  • qecore-headless "behave -kt automated_test" - This script takes an argument of a script to run.
    • In this case we run automated_test via behave. Bash is the default - in bash we can start behave on our own.
  • qecore-headless --session-type xorg - Start X11 session.
  • qecore-headless --session-type wayland - Start Wayland session.
  • qecore-headless --session-desktop gnome - Change desktop to GNOME.
  • qecore-headless --session-desktop gnome-classic - Change desktop to GNOME Classic.
  • qecore-headless --display - Set a DISPLAY number for the session, default is :0.
  • qecore-headless --dont-start - Do not start GDM.
  • qecore-headless --dont-kill - Do not kill GDM.
  • qecore-headless --restart - Restart running GDM
  • qecore-headless --keep X - Keep GDM alive for X tests, restart it after.
  • qecore-headless --keep-max - Keep GDM alive for as long as possible when testing - restart on test fail.
  • qecore-headless --force - Force will check configuration match and fails if for example X11 is started instead of Wayland.
  • qecore-headless --debug - Sets environment variable for dogtail, upon which dogtail will be logged.
  • qecore-headless --no-color - No colors for headless output.
  • qecore-headless --virtual-monitors X - Experimental option to enable virtual monitors for multi-monitor setup testing.
  • qecore-headless --enable-subscription - Do not disable subscription manager (disabled by default).
  • qecore-headless --help - for more information.

You can of course combine any of these together, if it makes sense.

Usage with behave.

The behave expected structure is very simple, we adjust it a bit for better readability:

features
├── environment.py
├── scenarios
|   └── main.feature
└── steps
    └── steps.py

The main file that qecore interacts with is environment.py.

behave has a few hooks that we can use to start our automation stack - lets look at before_all

from qecore.sandbox import TestSandbox

def before_all(context) -> None:
    """
    This function will be run once in every 'behave' command called.
    """

    # This is the most important part, initiation of TestSandbox.
    # Most of the setup happens in this class.
    context.sandbox = TestSandbox("gnome-terminal", context=context)

    # To define an application for testing, use `get_application` from sandbox.
    # If application is not installed as rpm but as a flatpak user can use `get_flatpak`
    # Attention: Do not define gnome-shell as an application.
    # The gnome-shell accessibility tree is available for you in `context.sandbox.shell`
    context.terminal = context.sandbox.get_application(
        name="gnome-terminal",
        a11y_app_name="gnome-terminal-server",
        desktop_file_name="org.gnome.Terminal.desktop",
    )
    # To modify the application with some functionality or work around some issues.
    # Most of the time no changes should be needed.
    context.terminal.exit_shortcut = "<Ctrl><Shift><Q>"

Let's look at before_scenario next.

The before_scenario will take care of every setup we need, to modify the execution, change the TestSandbox attributes after initiating the class.

All the attributes are set to values that were proven over the years of development. Some corner cases always exist though, so everything is customizable/hackable to enable a large variety of options.

def before_scenario(context, scenario) -> None:
    """
    This function will be run before every scenario in 'behave' command called.
    """
    context.sandbox.before_scenario(context, scenario)

To see everything it does, you can look at before_scenario implementation in documentation page.

And a final part after_scenario.

The after_scenario will take care of teardown. It will take care of anything that was set in before_scenario like stopping videos, fetching logs for our reports, uploading data to our HTML logs and closing any application started so that the session is cleared for the next test.

def after_scenario(context, scenario) -> None:
    """
    This function will be run after every scenario in 'behave' command called.
    """
    context.sandbox.after_scenario(context, scenario)

To see everything it does, you can look at after_scenario implementation in documentation page.

In the full example you can also see try/except usage. That is for recovery. Some issues can be fixed while running. We also need a way to end gracefully so that our data is loaded to the HTML page and not thrown away in case of a problem.

Working in steps.py.

After the setup in environment.py we can now start working on our automation in steps.py

The setup is important for two reasons.

• Now the test has all machine setup available through the sandbox, so they can query attributes or use sandbox methods that are very useful for testing.
  • context.sandbox.<attribute>
  • context.sandbox.<method>
• Users now do not have to start the applications and load accessibility tree to use, as they were defined in environment.py
  • In main.feature users can now use for example * Start application "terminal" via "command"
    • Application will start.
    • Load its accessibility tree.
    • Mark the application for cleanup after the test.
    • In steps.py users now have context.terminal.instance which is accessibility tree root of gnome-terminal-server so any dogtail query to Atspi will now work and automation can begin.

TestSandbox attributes.

Many attributes are present that can modify the test execution run. Some attributes and their default values that you can freely modify are:

• self.record_video = True - enable screencasting.
• self.opt_in_tree_on_fail = False - embed tree representation of a11y nodes in html report.
• self.status_report = True - adding defines rpm versions for quick check to html report.
• self.workspace_return = False - return to home workspace with shortcut <Super><Home>.
• self.set_keyring = True - set an empty keyring so that no dialog pops up during testing.
• self.wait_for_stable_video = True - waiting for video file to stop changing sizes.
• self.production = True - this value when set to False disables all embeds, screenshots, screencast.
• self.package_list = {"gnome-shell", "mutter", component} - modifiable set of rpms to check.
• self.default_application = None - first defined application in environment is default one.
• self.attach_coredump = False - generate and attach backtrace from coredump to html report on fail.
• self.attach_coredump_on_pass = False - generate and attach backtrace from coredump to html report on pass.
• self.attach_video = True - attach video to html report on fail.
• self.attach_video_on_pass = False - attach video to html report on pass.
• self.attach_journal = True - attach journal to html report on fail.
• self.attach_journal_on_pass = False - attach journal to html report on pass.
• self.attach_screenshot = True - attach screenshot to html report on fail.
• self.attach_screenshot_on_pass = False - attach screenshot to html report on pass.
• self.attach_faf = True - attach FAF to html report on fail.
• self.attach_faf_on_pass = True - attach FAF to html report on pass.
• self.compress_screenshot_on_pass = False - Converts all images into jpeg to save space on pass.
• self.compression_quality = 70 - Sets the jpeg quality, 0 - lowest, 100 - highest quality.

There are others but changing them most often is not recommended.

Environment variables.

There is a lot of options we have to modify our runs without the need to change python code.

• RICH_TRACEBACK=true behave... - to enable rich Traceback for debugging of issues.
• AUTORETRY=X behave... - give an automation test X tries to pass before it is marked as a FAIL.
  • This option is also available as a tag in feature files @autoretry=X
• STABILITY=X behave... - run an automation test X times and every try has to pass for it to be marked as a PASS.
  • This option is also available as a tag in feature files @stability=X
• QECORE_EMBED_ALL=true behave... - all available data is embedded even if the suite PASSED (We do not embed on PASS by default).
• QECORE_NO_CACHE=true behave... - do not keep cache and regenerate files.
• QECORE_ENABLE_SCREENCAST=true behave... - disable screencasting.
• LOGGING=true behave... - enables logging, this is a very verbose log about qecore execution.
• PRODUCTION=false behave... - disables all embedding, video recording, screenshot capture.
• BACKTRACE=true behave... - enables backtrace fetching from coredumpctl list.

You are able to use multiple variables, if the combination makes sense, together.

Hacking.

From everything that I showed here, everything is customizable to some degree.

I implemented most the methods in a way that changing how they operate is done easily from environment.py

In case you need some specific change, look through the code and you should be able to identify what needs to be done (everything is in one place really, sandbox, before_scenario, after_scenario, that's it, everything else is called from there)

This project was featured in Fedora Magazine:

• https://fedoramagazine.org/automation-through-accessibility/
• Full technical solution of our team's (Red Hat DesktopQE) automation stack https://modehnal.github.io/

Execute unit tests

NOTE: Tests need to be updated for Fedora 40 and newer and RHEL-10 and newer.

Execute the tests (from the project root directory) on machine with dogtail:

rm -f /tmp/qecore_version_status.txt
rm -f dist/*.whl
python3 -m build
python3 -m pip install --force-reinstall --upgrade dist/qecore*.whl
sudo -u test scripts/qecore-headless "behave -f html-pretty -o /tmp/report_qecore.html -f plain tests/features"

You can use -f pretty instead of -f plain to get colored output.

The standard output should not contain any python traceback, produced HTML should be complete (after first scenario there is Status).