Xumes

Xumes is a game testing tool designed to be used on top of games without the need to modify the game sources. It provides a framework for observing and interacting with game elements, training models, and conducting tests. For the moment, only the Godot framework has an implemented module for interacting with Xumes. So for the moment, the documentation will be focused on the implementation with Godot. Please check the documentation for more information.

Framework

The framework is based on the BDD methodology, that's means that two type of files are needed to be write for conducting tests. Those file are the feature files and the steps files.

The following diagram illustrates the architecture of the framework:

Installation

pip install xumes

Using with Godot

To use Xumes with Godot, you need to compile Godot from sources. Follow the instructions in the Godot documentation <https://docs.godotengine.org/en/stable/development/compiling/index.html>_.

Clone the following repository instead of the official one:

git clone https://github.com/mastainvin/godot.git

Don't forget to checkout to the xumes branch before build Godot.

To start the Godot editor, run the following command:

 ./bin/godot.x11.tools.64 --editor --path <path_to_godot_project>

Quick start

Once Xumes and Godot are installed, you can test your installation with the Flappy Bird example. First, clone the repository:

git clone https://github.com/mastainvin/Xumes-Examples.git

Then, navigate to the FlappyBird directory:

cd Xumes-Examples/Godot/flappy_bird/

Run the tests with the following command:

python -m xumes test ./Tests/PipeSizeTest/features -h 10 -i 100

⚠️ Always use python -m xumes to run the tests. This ensures the use Xumes as a module and be able to import other modules in the project.

This command will run the tests found in the features directory in headless mode (without rendering the game), with a frame rate limit of 10 FPS and 100 iterations.

You should see the following output:

               TEST REPORT

9 tests passed on a total of 9 in 00:01:16.
Tests passed:
    - PipeSize/Testing the pipe size [2] - (i=0.5, j=1, k=2)
    - PipeSize/Testing the pipe size easy [1] - (i=0.5, j=0.5, k=2)
    - PipeSize/Testing the pipe size easy [2] - (i=1, j=1, k=2)
    - PipeSize/Testing the pipe size hard [1] - (i=1, j=0, k=2)
    - PipeSize/Testing the pipe size [1] - (i=0.5, j=0, k=2)
    - PipeSize/Testing the pipe size easy [0] - (i=0, j=0, k=2)
    - PipeSize/Testing the pipe size hard [0] - (i=0, j=1, k=2)
    - PipeSize/Testing the pipe size [0] - (i=0, j=0.5, k=2)
    - PipeSize/Testing the pipe size [3] - (i=1, j=0.5, k=2)

Here is the video of godot first build and test execution.