Skip to main content

A framework for generating test suites from external configuration files.

Project description


The pyutilib.autotest package provides a facility for automatically configuring tests that are executed with Python’s unittest package. This capability is tailored for tests where one or more solvers are applied to one or more problems. This testing structure is particularly useful for evaluating the execution of external executables on datasets.

There are three main steps for configuring and applying pyutilib.autotest:

  1. Create a configuration file
  2. Create a test driver plugin
  3. Setup the Python module that will be used to apply the tests

These steps are described in the following subsections.

Creating a Configuration File

Currently, pyutilib.autotest only supports a YAML configuration file. The top-level collection in this configuration is a mapping with the following keys:

This section contains a list of Python expressions that are executed while setting up the test suites.
This section specifies the name of the test driver that is used to execute tests.
This section contains a mapping of solver names to solver options.
This section contains a mapping of problem names to problem options.
This section contains a mapping of test suite names to suite configurations.

The following example illustrates the structure of a YAML configuration file. (This example is the file pyutilib.autotest/examples/autotest/example1.yml in the pyutilib.autotest distribution.)

driver: example

    - import example

        name: cat
        cat_options: -n

        file: README.txt
        file: LoremIpsum1.txt


            - smoke
            - suite1

            - nightly
            - suite2
            - solver: cat
              problem: p1
            - solver: cat2
              problem: p2
            - solver: echo
              problem: p3

            - suite3
                solver: cat
                cat_options: -n

The test driver example is defined in the file, which is imported with the directives in the python section.

Within the solvers and problems sections, each solver and problem can specify additional options that are passed into the test. Note that these options are not distinguished for the test; option name conflicts will results in unpredictable behavior.

Three solvers are defined in this example, which apply the unix cat and echo commands. By default, the name of the solver is assumed to be the name of the key for the solver map. Solver cat2 illustrates how a solver name can be specified separately from the solver key.

The suites section defines one or more test suites. Each test suite consists of a mapping with the following sections:

This optional section contains a list of strings that are solver categories. These categories can be used to select characteristics of the test suites that are executed.
This section contains a mapping of solvers. Note that additional options can be provided here to further customize the solver behavior. By default, the solver name is assumed to be the key value, but the solver key can be used to explicitly define the solver that is associated with a customized solver mapping.
This section contains a list of problems. Note that additional options can be provided here to further customize the test behavior. By default, the problem name is assumed to be the key value, but the problem key can be used to explicitly define the problem that is associated with a customized problem mapping.
This optional section contains a list of solver-problem pairs, which defines the actual tests that are defined. Note that the solver and problem values map to the keys defined in the solvers and problems sections of a test suite; thus, this section can use the names of customized solvers and problems that are not defined in the top-level solvers and problems sections. Finally, if this section is omitted, then all combinations of solvers and problems are used to create tests.

In this example, three suites are defined to illustrate different features of the test driver:

A simple test suite in which all solvers are applied to a single problem.
The tests section is specified to select a subset of all combinations of solvers and problems to test.
The catx solver is customized from cat to operate like the cat2 solver.

Creating a Test Driver

The test configuration used by pyutilib.autotest is quite generic. It specifies what combinations of solvers and problems are to be tested, along with their corresponding options. However, it does not specify how the test is performed. This is done by a test driver class.

In general, test driver classes are required to be plugins that can be dynamically created by pyutilib.autotest to execute tests. The easiest way to define a test driver plugin is to inherit from the TestDriverBase class. For example, the following test driver is used in by the earlier test configuration; this plugin is defined in pyutilib.autotest/examples/autotest/

import pyutilib.autotest
from pyutilib.component.core import *
import pyutilib.subprocess

class ExampleTestDriver(pyutilib.autotest.TestDriverBase):
    This test driver executes a unix command and compares its output
    with a baseline value.


    def run_test(self, testcase, name, options):
        """Execute a single test in the suite"""
        name = options.suite+'_'+name
        cmd = options.solver+' '
        if not options.cat_options is None:
            cmd += options.cat_options+' '
        cmd += options.file
        print "Running test suite '%s'  test '%s'  command '%s'" % \
                (options.suite, name, cmd), outfile=options.currdir+'test_'+name+".out")

The alias function is used to specify the name of this plugin; this is the name of the test driver used in the test configuration file.

The run_test method executes a single test in the test suite. Note that this method is passed in testcase, which is the test suite class. Thus, this method can directly apply the unittest methods that are defined in this class (e.g. assertEquals).

In this example, a unix command-line is create from the solver name, the solver options, and the problem filename. This is executed with the function, which redirects output to a log file. This log file is then compared with a baseline file using the failUnlessFileEqualsBaseline, which is defined in the unittest extensions in

Note that a variety of other standard unit test methods can also be defined by this test driver. This driver is a ITestDriver plugin, and the API for this plugin is:

class ITestDriver(Interface):

    def setUpClass(self, cls, options):
        """Set-up the class that defines the suite of tests"""

    def tearDownClass(self, cls, options):
        """Tear-down the class that defines the suite of tests"""

    def setUp(self, testcase, options):
        """Set-up a single test in the suite"""

    def tearDown(self, testcase, options):
        """Tear-down a single test in the suite"""

    def run_test(self, testcase, name, options):
        """Execute a single test in the suite"""

Creating a Test Module

Virtually all of the work needed to create test suites is automated by pyutilib.autotest. The following test module is used in this example; (see pyutilib.autotest/examples/autotest/

import os
import sys
from os.path import abspath, dirname
currdir = dirname(abspath(__file__))+os.sep

import as unittest
import pyutilib.autotest

if __name__ == "__main__":
    pyutilib.autotest.create_test_suites(filename=currdir+'example1.yml', _globals=globals())

The first four lines are needed to identify the current directory, where the test configuration file resides.

Note that is imported as unittest, which reminds the user that this is a unittest extension package. (Specifically, this package contains hooks needed to dynamically add functions as test methods in test suites.)

The pyutilib.autotest packages is imported so the create_test_suites function can be executed. The arguments to this function are the test configuration file, and the global dictionary.

Finally, unittest.main() is executed, as in any unittest module. Tests can be executed using standard unittest command-line options. One extension to this behavior is the use of the PYUTILIB_AUTOTEST_CATEGORIES or PYUTILIB_UNITTEST_CATEGORIES environmental variables; if one of these is specified, then pyutilib.autotest assumes that this data contains a comma-separated list of categories that are used to select the test suites that are constructed. Specifically, if a test suite contains one of the specified test categories, then it will be executed.

The pyutilib_test_driver Command

The pyutilib_test_driver command can be used to execute tests defined in a configuration file without creating a test module. In practice, test modules are typically needed to support test discovery with tools like nose. However, this command provides several features that are useful when diagnosing tests.

The command-line behavior of pyutilib_test_driver extends the API of unittest.main(). The following additional options are provided to allow the user to interrogate the tests that are defined by the test configuration file:

--help-suites Print the test suites that can be executed
 Print the tests in the specified test suite
 Print the test suite categories that can be specified


BSD. See the LICENSE.txt file.


Third Party Software


Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
pyutilib.autotest-1.4.1.tar.gz (19.7 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page