Simple command line tool for executing Containers Testing Framework
Containers Testing Framework (CTF) is a simple wrapper around Behave testing framework. Behave is a promising approach for testing containers, since it enables one to focus on describing the behavior of the container from a High Level and in simple English. For more information on how to use Behave and how to write tests using Behave, please refer to the Behave project page.
CTF tries to re-use the remote execution of steps model from UATFramework so the tests can be executed locally or on a remote machine or VM. The support is still in progress.
If you want to implement project-specific tests, you should create the following directory structure inside your project directory my_project_dir:
my_project_dir/ tests/ features/ my_cool.feature my_other_cool.feature steps/ my_cool_steps.py my_other_cool_steps.py environment.py tests.conf
The best way to leverage the framework is to implement Features and Steps that are common for a set of containers as a remote test and then include it in the testing configuration of all containers.
Place any features and scenarios, specific for your particular container inside this directory. Your scenarios can use any of the steps implemented inside tests/steps directory or steps from any remote test you listed inside the tests.conf file. CTF CLI tool will combine all these steps together, so that Behave is able to find them when run.
Place any steps that are specific for your features inside this directory. To make sure the steps can be executed on the remote machine or locally, always use context.run() for running any commands. In the background, the command will be run locally or on remote machine, based on the CLI configuration.
The CLI tool passes some runtime arguments to Behave, when executing it. The values are available through the context as context.config.userdata dictionary like object. To learn more about it, please read the Behave documentation. The following values are passed:
This option contains the absolute path to the Dockerfile wich should be tested. It is always set.
This option contains the name of the image to test. It is passed and set only if the name was passed to CLI tool using -i option.
The tests.conf file contains configuration telling the CTF what extra remote tests (Steps and Features) to include when testing the container using Behave. It is a simple INI file. Each remote test needs to have its own section and inside the section specify Steps and Features options. These has to point to remote git repositories containing the appropriate Steps and Features. Example of tests.conf:
[common-tests] Steps=https://github.com/Containers-Testing-Framework/common-steps.git Features=https://github.com/Containers-Testing-Framework/common-features.git
You can implement any of the methods that are typically used with Behave inside this file. They will be combined with the CTF common environment.py file. And your methods will be invoked before the CTF hooks.
git clone https://github.com/Containers-Testing-Framework/ctf-cli.git
Optional: clone an example repository
git clone https://github.com/Containers-Testing-Framework/example-project-postgresql.git
Change to ctf-cli directory
Install python dependencies
[sudo] pip install -r requirements.txt
Copy sample configuration files and edit as necessary
cp ctf.conf.sample ctf.conf cp tests.conf.sample tests.conf
Change to project directory and get common features
cd example-project-postgresql ../ctf-cli/ctf-cli.py remote add features https://github.com/Containers-Testing-Framework/common-features.git
The key part of the framework is the CLI tool called ctf-cli. It gathers information, reads configurations, sets up the environment for Behave and runs it. Currently ctf-cli tool supports only ansible for running commands on some host. To run your tests make sure you included configuration for ansible in the ctf-cli configuration.
You can pass the path to the configuration file on the command line using the -c or --cli-config option. If not provided the tool tries to find the configuration in this order of preference:
The configuration must include configuration for ansible. The setup on the host you want to use needs to be done manually before running the ctf-cli (e.g. setting up ssh keys, etc.) An example of the configuration for ansible inside the ctf.conf can look like this:
[ansible] Host=127.0.0.1 User=root Method=ssh
The ctf-cli should be executed inside the directory of the project, containing the Dockerfile. There are couple of options one can pass to the CLI tool:
When ctf-cli is executed in my_proj_dir project directory the following happens: