python-tool-competition-2024 0.1.1

Pipeline to create test generators

Python Tool Competition Runner

Usage

Installation

This tool requires at least Python 3.9 and poetry.

pip install python-tool-competition-2024

Creating a New Test Generator Project

Run python-tool-competition-2024 init and follow the instructions on the screen. During the setup process, you need to provide the following information:

• Human-readable name of the test generator (mandatory)
• Directory in which the project will be created (by default: the current directory)
• Your full name (mandatory)
• Your email address (mandatory)
• Information required to create pyproject.toml:
  • Project's version (by default 0.1.0)
  • Author (by default, it will use the information provided in the prior steps)
  • Licence (optional)
  • Compatible Python versions for your generator
  • Project's main dependencies (can be defined during the setup or afterward)
  • Project's development dependencies (can be defined during the setup or afterward)

This will create a poetry project with all required files. It creates a template of the generator and exposes it as a plugin of the competition runner.

Additionally, some very basic example targets are created as well. You can add your own files to the targets directory.

Now you can implement the build_test function of your generator. This will be called for each target file and it expects that either a TestGenerationSuccess or TestGenerationFailure is returned. The success contains the body of the generated test file. Storing the file is handled by the runner that runs build_test. The failure contains a reason and lines that describe the failure.

For examples see:

• https://github.com/ThunderKey/python-tool-competition-2024-klara
• https://github.com/ThunderKey/python-tool-competition-2024-hypothesis-ghostwriter

Running inside of the Project

Inside of the create project run poetry run python-tool-competition-2024 run <generator name>. This will start the test generation process.

With poetry run python-tool-competition-2024 run -h you can find out what generators were detected.

The tool does not only execute the test generator, it also runs the generated tests against the code to measure different metrics: it measures line and branch coverage using the coverage framework; it furthermore computes mutation score utilizing the cosmic-ray tool.

After finishing the test generation process, the script will print the information regarding the coverage achieved by the tests generated by your tool. It will also save these data as XML files in the <generator name>/coverages directory.

Improving the Competition Runner

• Installation: poetry install
• Testing: tox
• Use pre-commit if possible.