DTest 0.5.0

Dependency-based Threaded Test Framework

The DTest framework is a testing framework, similar to the standard unittest package provided by Python. The value-add for DTest, however, is that test execution is threaded, through use of the eventlet package. The DTest package also provides the concept of “dependencies” between tests and test fixtures–thus the “D” in “DTest”–which ensure that tests don’t run until the matching set up test fixtures have completed, and that the tear down test fixtures don’t run until all the associated tests have completed. Dependencies may also be used to ensure that tests requiring the availability of certain functionality don’t run if the tests of that specific functionality fail.

Writing Tests

The simplest test programs are simple functions with names beginning with “test,” located in Python source files whose names also begin with “test.” It is not even necessary to import any portion of the DTest framework. If tests are collected in classes, however, or if use of the more advanced features of DTest is desired, a simple from dtest import * is necessary. This makes available the DTestCase class–which should be extended by all classes containing tests–as well as such decorators as @skip and @nottest.

Tests may be performed using the standard Python assert statement; however, a number of utility routines are available in the dtest.util module (also safe for import *). Many of these utility routines have names similar to methods of unittest.TestCase–e.g., dtest.util.assert_dict_equal() is analogous to unittest.TestCase.assertDictEqual().

Test Fixtures

The DTest framework supports test fixtures–set up and tear down functions–at the class, module, and package level. Package-level fixtures consist of functions named setUp() and tearDown() contained within “__init__.py” files; similarly, module-level fixtures consist of functions samed setUp() and tearDown() within modules containing test functions and classes of test methods. At the class level, classes may contain setUpClass() and tearDownClass() class methods (or static methods), which may perform set up and tear down for each class. In all cases, the setUp() functions and the setUpClass() method are executed before any of the tests within the same scope; similarly, after all the tests at a given scope have executed, the corresponding tearDownClass() method and tearDown() functions are executed.

The DTest framework also supports per-test setUp() and tearDown() functions or methods, which are run before and after each associated test. For classes containing tests, each test automatically has the setUp() and tearDown() methods of the class associated with them; however, for all tests, these fixtures can be explicitly set (or overridden from the class default). Consider the following example:

@istest
def test_something():
    # Test something here
    pass

@test_something.setUp
def something_setup():
    # Get everything set up ready to go...
    pass

@test_something.tearDown
def something_teardown():
    # Clean up after ourselves
    pass

In this example, a DTest decorator (other than @nottest) is necessary preceding test_something(); here we used @istest, but any other available DTest decorator could be used here. This makes the @test_something.setUp and @test_something.tearDown decorators available. (For something analogous in the standard Python, check out the built-in @property decorator.)

Test Resources

Many test suites use test fixtures to set up temporary resources needed for a particular test. For instance, it’s not uncommon for a fixture to set up a utility object, such as a server client, which could be reused by other tests. The DTest framework provides an alternative means of setting up such objects: test resources.

A test resource is any single object that may be required by a given test. To create one, set up a class extending the Resource class and implement the setUp() method on the class (and, optionally, the tearDown() method). There are two additional class attributes that can be set. The first is oneshot: if set to True, the resource returned by setUp() will only be used once, then discarded. The second is dirtymeths, which should contain a list of methods which, when called, will cause the object to become “dirty”, causing it to be discarded after the test. (Setting or deleting object attributes will also cause the object to be marked as “dirty”.)

To mark that a test requires a particular resource, use the @require() decorator; this decorator takes keyword arguments, where the keys will be taken as the names of arguments to the test, and the values must be instances of subclasses of Resource. When the test is run, DTest will create the actual resource objects and pass them to the test as keyword arguments. As long as the object does not become dirty, it will be reused for subsequent tests, subject to threading constraints (resource objects may only be used by one thread at a time).

Resource Limitations

Resources are subject to one limitation: the object actually passed to the test is a proxy object which delegates attribute accesses to the actual object allocated by the setUp() method. Because of optimizations within Python itself, it is not possible for this proxy object to properly delegate special methods, such as __getitem__() or __add__(). Because of this, it is possible to retrieve the true resource object, using the getobject() function. Because this removes the ability of the resources system to determine if the resource becomes dirty, the dirty() and clean() functions are also provided. Finally, to prevent an access from marking the object as dirty, the cleanaccess() function can be used in conjunction with the with statement like so:

with cleanaccess(resource):
    resource.attribute = "some value"

Without the with statement, this attribute setting would cause the resource to be marked as dirty, but the with inhibits this. Note that it is legal to nest calls to cleanaccess(), if necessary.

Resource Options

Resources may be specified with options, which should be string-coercible constants. Any positional or keyword arguments passed to the Resource constructor will be saved and passed to the setUp() method when a resource must be constructed. In addition, the resource caching mechanism uses these options to ensure that a test is only passed resources with matching options.

Optional tearDown()

If some special cleanup is needed for a resource, implement the tearDown() method on your Resource subclass. It should take two arguments: the object that was returned by setUp(), and the status of the test. For most resources, unless a test renders them dirty, the status will be None, and tearDown() will be called after all tests have run to completion; however, for resources which have oneshot set to True, the status should never be None. One possible use case for this is a test which uses temporary files, which should be cleaned up after the test passes; should the test fail, it may be useful to leave the temporary file around for debugging purposes.

Running Tests

Running tests using the DTest framework is fairly straight-forward. A script called run-dtests is available. By default, the current directory is scanned for all modules or packages whose names begin with “test”; the search also recurses down through all packages. (A “package” is defined as a directory containing “__init__.py”.) Once all tests are discovered, they are then executed, and the results of the tests emitted to standard output.

Several command-line options are available for controlling the behavior of run-dtests. For instance, the “–no-skip” option will cause run-dtests to run all tests, even those decorated with the @skip decorator, and the “-d” option causes run-dtests to search a specific directory, rather than the current directory. For a full list of options, use the “-h” or “–help” option.

Running run-dtests from the command line is not the only way to run tests, however. The run-dtests script is a very simple script that parses command-line options (using the OptionParser constructed by the dtest.optparser() function), converts those options into a set of keyword arguments (using dtest.opts_to_args()), then passes those keyword arguments to the dtest.main() function. Users can use these functions to build the same functionality with user-specific extensions, such as providing an alternate DTestOutput instance to control how test results are displayed, or providing an alternate method for controlling which tests are skipped. See the documentation strings for these functions and classes for more information.