A framework for generating test suites from external configuration files.
The pyutilib.autotest package provides a facility for automatically configuring tests that are executed with Python’s unittest package. This capability is tailored for tests where one or more solvers are applied to one or more problems. This testing structure is particularly useful for evaluating the execution of external executables on datasets.
There are three main steps for configuring and applying pyutilib.autotest:
These steps are described in the following subsections.
Currently, pyutilib.autotest only supports a YAML configuration file. The top-level collection in this configuration is a mapping with the following keys:
The following example illustrates the structure of a YAML configuration file. (This example is the file pyutilib.autotest/examples/autotest/example1.yml in the pyutilib.autotest distribution.)
driver: example python: - import example solvers: cat: cat2: name: cat cat_options: -n echo: problems: p1: file: README.txt p2: file: example.py p3: file: LoremIpsum1.txt suites: suite1: categories: - smoke - suite1 solvers: cat: cat2: echo: problems: p1: suite2: categories: - nightly - suite2 solvers: cat: cat2: echo: problems: p1: p2: p3: tests: - solver: cat problem: p1 - solver: cat2 problem: p2 - solver: echo problem: p3 suite3: categories: - suite3 solvers: cat: catx: solver: cat cat_options: -n problems: p1:
The test driver example is defined in the example.py file, which is imported with the directives in the python section.
Within the solvers and problems sections, each solver and problem can specify additional options that are passed into the test. Note that these options are not distinguished for the test; option name conflicts will results in unpredictable behavior.
Three solvers are defined in this example, which apply the unix cat and echo commands. By default, the name of the solver is assumed to be the name of the key for the solver map. Solver cat2 illustrates how a solver name can be specified separately from the solver key.
The suites section defines one or more test suites. Each test suite consists of a mapping with the following sections:
In this example, three suites are defined to illustrate different features of the test driver:
The test configuration used by pyutilib.autotest is quite generic. It specifies what combinations of solvers and problems are to be tested, along with their corresponding options. However, it does not specify how the test is performed. This is done by a test driver class.
In general, test driver classes are required to be plugins that can be dynamically created by pyutilib.autotest to execute tests. The easiest way to define a test driver plugin is to inherit from the TestDriverBase class. For example, the following test driver is used in by the earlier test configuration; this plugin is defined in pyutilib.autotest/examples/autotest/example.py.
import pyutilib.autotest from pyutilib.component.core import * import pyutilib.subprocess class ExampleTestDriver(pyutilib.autotest.TestDriverBase): """ This test driver executes a unix command and compares its output with a baseline value. """ alias('example') def run_test(self, testcase, name, options): """Execute a single test in the suite""" name = options.suite+'_'+name cmd = options.solver+' ' if not options.cat_options is None: cmd += options.cat_options+' ' cmd += options.file print "Running test suite '%s' test '%s' command '%s'" % \ (options.suite, name, cmd) pyutilib.subprocess.run(cmd, outfile=options.currdir+'test_'+name+".out") testcase.failUnlessFileEqualsBaseline( options.currdir+'test_'+name+".out", options.currdir+'test_'+name+".txt")
The alias function is used to specify the name of this plugin; this is the name of the test driver used in the test configuration file.
The run_test method executes a single test in the test suite. Note that this method is passed in testcase, which is the test suite class. Thus, this method can directly apply the unittest methods that are defined in this class (e.g. assertEquals).
In this example, a unix command-line is create from the solver name, the solver options, and the problem filename. This is executed with the pyutilib.subprocess.run function, which redirects output to a log file. This log file is then compared with a baseline file using the failUnlessFileEqualsBaseline, which is defined in the unittest extensions in pyutilib.th.
Note that a variety of other standard unit test methods can also be defined by this test driver. This driver is a ITestDriver plugin, and the API for this plugin is:
class ITestDriver(Interface): def setUpClass(self, cls, options): """Set-up the class that defines the suite of tests""" def tearDownClass(self, cls, options): """Tear-down the class that defines the suite of tests""" def setUp(self, testcase, options): """Set-up a single test in the suite""" def tearDown(self, testcase, options): """Tear-down a single test in the suite""" def run_test(self, testcase, name, options): """Execute a single test in the suite"""
Virtually all of the work needed to create test suites is automated by pyutilib.autotest. The following test module is used in this example; (see pyutilib.autotest/examples/autotest/autotest.py):
import os import sys from os.path import abspath, dirname currdir = dirname(abspath(__file__))+os.sep import pyutilib.th as unittest import pyutilib.autotest if __name__ == "__main__": pyutilib.autotest.create_test_suites(filename=currdir+'example1.yml', _globals=globals()) unittest.main()
The first four lines are needed to identify the current directory, where the test configuration file resides.
Note that pyutilib.th is imported as unittest, which reminds the user that this is a unittest extension package. (Specifically, this package contains hooks needed to dynamically add functions as test methods in test suites.)
The pyutilib.autotest packages is imported so the create_test_suites function can be executed. The arguments to this function are the test configuration file, and the global dictionary.
Finally, unittest.main() is executed, as in any unittest module. Tests can be executed using standard unittest command-line options. One extension to this behavior is the use of the PYUTILIB_AUTOTEST_CATEGORIES or PYUTILIB_UNITTEST_CATEGORIES environmental variables; if one of these is specified, then pyutilib.autotest assumes that this data contains a comma-separated list of categories that are used to select the test suites that are constructed. Specifically, if a test suite contains one of the specified test categories, then it will be executed.
The pyutilib_test_driver command can be used to execute tests defined in a configuration file without creating a test module. In practice, test modules are typically needed to support test discovery with tools like nose. However, this command provides several features that are useful when diagnosing tests.
The command-line behavior of pyutilib_test_driver extends the API of unittest.main(). The following additional options are provided to allow the user to interrogate the tests that are defined by the test configuration file:
|--help-suites||Print the test suites that can be executed|
|Print the tests in the specified test suite|
|Print the test suite categories that can be specified|
BSD. See the LICENSE.txt file.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|File Name & Checksum SHA256 Checksum Help||Version||File Type||Upload Date|
|pyutilib.autotest-2.0.1-py2-none-any.whl (19.8 kB) Copy SHA256 Checksum SHA256||2.7||Wheel||May 26, 2014|
|pyutilib.autotest-2.0.1.tar.gz (19.4 kB) Copy SHA256 Checksum SHA256||–||Source||Feb 25, 2013|