Skip to main content

Easier test setup for Zope 3 projects and other Python packages.

Project description

z3c.testsetup

is a package that provides some convenience stuff to enable rapid test setup for Zope 3 components and normal Python packages. Currently doctests (normal unit doctests and functional doctests) and usual Python tests made of unittest.TestCase definitions are supported.

Doctests and test modules are found throughout a whole package and registered with sensible, modifiable defaults.

Also support for reusable test setups is provided by so-called TestGetters and TestCollectors.

Setting up doctests (contrary to writing those tests) can become cumbersome with Zope 3. In the environment you often have to prepare complex things like test layers, setup functions, teardown functions and much more. Often this steps have to be done again and again.

z3c.testsetup can shorten this work by setting sensible defaults for the most important aspects of test setups.

See README.txt in the src/z3c/testsetup directory for API documentation. There is also extensive documentation for (hopefully) every aspect of the package in this directory. See all the .txt files to learn more.

Note, that this is alphaware! Do not use it in productive environments!

Prerequisites

You need:

Other needed packages will be downloaded during installation. Therefore you need an internet connection during installation.

Installation

From the root of the package run:

$ python2.4 bootstrap/bootstrap.py

This will download and install everything to run buildout in the next step. Afterwards an executable script buildout should be available in the freshly created bin/ directory.

Next, fetch all needed packages, install them and create provided scripts:

$ bin/buildout

This should create a test script in bin/.

Running:

$ bin/test

you can test the installed package.

Usage

See README.txt and the other .txt files in the src/z3c/testsetup directory for API documentation.

Detailed Documentation

z3c.testsetup

Easy testsetups for Zope 3 and Python projects.

Setting up tests for Zope 3 projects sometimes tends to be cumbersome. z3c.testsetup jumps in here, to support much flatter test setups. The package supports three kinds of tests:

  • normal python tests: i.e. tests, that consist of python modules which in turn contain unittest.TestCase classes.
  • unit doctests: i.e. tests, that are written as doctests, but require no complicated layer setup etc.
  • functional doctests: i.e. tests, that are written as doctests, but also require a more or less complex framework to test for example browser requests.

z3c.testsetup is package-oriented. That means, it registers more or less automatically all the three kinds of tests mentioned above insofar they are part of a certain package.

This is a general introduction to z3c.testsetup. For setup examples you might see the cave package contained in the tests/ directory. More details on special topics can be found in the appropriate .txt files in this directory.

Basic Example

The shortest test setup possible with z3c.testsetup looks like this:

>>> import z3c.testsetup
>>> test_suite = z3c.testsetup.register_all_tests(
...                   'z3c.testsetup.tests.cave')

It is sufficient to put this lines into a python module which is found by your testrunner (see samplesetup_short examples in the cave package and testrunner.txt).

To sum it up, testsetup with z3c.testsetup is done in two steps:

  1. Make sure your testfiles are named properly (.txt/.rst for doctests, valid python modules for usual unit tests) and provide a suitable marker string as explained below (How to mark testfiles/modules).

  2. Write a test setup module which is named so that your testrunner finds it and in this module call:

    test_suite = z3c.testsetup.register_all_tests(<package>)
    

    where <package> must be a package object. Instead of a package object you can also pass the package’s dotted name as string like ‘z3c.testsetup.tests.cave’.

Given that, this setup should find all doctests (unit and functional) as well as python tests in the package and register them.

Customized Setups

The register_all_tests function mentioned above accepts a bunch of keyword parameters:

register_all_tests(pkg_or_dotted_name, filter_func, extensions,
                   encoding, checker,
                   globs, setup, teardown, optionflags
                   zcml_config, layer_name, layer)

where all but the first parameter are keyword paramters and all but the package parameter are optional.

While filter_func and extensions determine the set of testfiles to be found, the other paramters tell how to setup single tests.

  • filter_func (ufilter_func, ffilter_func)

    a function that takes an absolute filepath and returns True or False, depending on whether the file should be included in the test suite as doctest or not. filter_func applies only to doctests.

    We setup a few things to check that:

    >>> import os
    >>> import unittest
    >>> suite = test_suite()
    >>> suite.countTestCases()
    4
    

    Okay, the callable in test_suite we created above with register_all_tests apparently delivered four testcases. This is normally also the number of files involved, but let’s check that correctly.

    We did setup a function get_basenames_from_suite in this testing environment (as a globs entry) which determines the basenames of the paths of all testcases contained in a TestSuite:

    >>> get_basenames_from_suite(suite)
    ['file1.py', 'file1.rst', 'file1.txt', 'subdirfile.txt']
    

    Ah, okay. There are in fact four files, in which testcases were found. Now, we define a plain filter function:

    >>> def custom_file_filter(path):
    ...     """Accept all txt files."""
    ...     return path.endswith('.txt')
    

    This one accepts all ‘.txt’ files. We run register_all_tests again, but this time with a filter_func parameter:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     filter_func=custom_file_filter)
    

    To get the resulting test suite, we again call the returned callable:

    >>> suite = test_suite()
    >>> get_basenames_from_suite(suite)
    ['file1.py', 'file1.txt', 'file1.txt', 'subdirfile.txt',
    'subdirfile.txt']
    

    Compared with the first call to register_all_tests we got some strange results here: there is a ‘.py’ file, which should have been refused by our filter function and the other two files appear twice. What happened?

    The python module is included, because python tests are not filtered by filter_func. Instead this value applies only to doctests.

    The second strange result, that every .txt file appears twice in the list, comes from the fact, that the filter is valid for unit and functional doctests at the same time. In other words: the tests in those .txt files are registered twice, as unittests and a second time as functional tests as well.

    If you want a filter function for functional doctests or unit doctests only, then you can use ffilter_func and ufilter_func respectively:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     ffilter_func=custom_file_filter,
    ...     ufilter_func=lambda x: False)
    
    >>> suite = test_suite()
    >>> get_basenames_from_suite(suite)
    ['file1.py', 'file1.txt', 'subdirfile.txt']
    

    As expected, every .txt file was only registered once. The same happens, when we switch and accept only unit doctests:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     ffilter_func=lambda x: False,
    ...     ufilter_func=custom_file_filter)
    
    >>> suite = test_suite()
    >>> get_basenames_from_suite(suite)
    ['file1.py', 'file1.txt', 'subdirfile.txt']
    

    If you specify both, a filter_func and a more specialized ufilter_func or ffilter_func, then this has the same effect as passing both, ufilter_func and ffilter_func:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     ffilter_func=lambda x: False,
    ...     filter_func=custom_file_filter)
    
    >>> suite = test_suite()
    >>> get_basenames_from_suite(suite)
    ['file1.py', 'file1.txt', 'subdirfile.txt']
    
  • pfilter_func:

    Does basically the same as the filter_funcs above, but handles Python modules instead of file paths. It therefore determines the set of ‘normal’ Python tests accepted and does not touch the set of doctests accepted.

    We define a simple custom filter:

    >>> def custom_module_filter(module):
    ...     return 'Tests with real' in str(module.__doc__)
    

    that checks for a certain string in modules’ doc strings.

    Now we start again with pfilter_func set:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     pfilter_func=custom_module_filter)
    >>> suite = test_suite()
    >>> get_basenames_from_suite(suite)
    ['file1.py', 'file1.rst', 'file1.txt', 'notatest2.py', 'subdirfile.txt']
    

    Because file1.py and notatest2.py in the cave package contain the required string, this is correct. Because the default function checks for the string :Test-Layer: python, the second module was omitted by default.

    Now let’s use a filter, that refuses all modules:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     pfilter_func=lambda x: False)
    >>> suite = test_suite()
    >>> get_basenames_from_suite(suite)
    ['file1.rst', 'file1.txt', 'subdirfile.txt']
    

    All Python modules vanished from the list.

    In case you wonder, why not all the other Python files of the cave package (__init__.py, for example) appear in one of the lists: we get only the result list, which contains only such modules, which provide unittest.TestCase definitions. Because most modules of the cave package don’t define test cases, they do not appear in the list. This automatism is driven by a unittest.TestLoader. See http://docs.python.org/lib/testloader-objects.html to learn more about test loaders.

  • extensions (uextensions, fextensions):

    a list of filename extensions to be considered during test search. Default value is [‘.txt’, ‘.rst’]. Python tests are not touched by this (they have to be regular Python modules with ‘.py’ extension).

    Note, that the extensions attribute is used by the default filter function. If you pass your own filter function using [u|f]filter_func, then the extensions filtering won’t work any more.

    If we want to register .foo files, we can do so:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     extensions=['.foo'])
    >>> suite = test_suite()
    >>> get_basenames_from_suite(suite)
    ['file1.py', 'notatest1.foo', 'notatest1.foo']
    

    Note, that only files that contain an appropriate marker are found, regardless of the filename extension. The new .foo file contains a marker for unit doctests and functional doctests, such it is included twice in the list.

    As we can see, the new file appears twice. This is, because it is registered as functional doctest and unitdoctest as well.

    To collect only functional doctests with a certain set of filename extensions you can use: fextensions:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     fextensions=['.foo'])
    >>> suite = test_suite()
    >>> get_basenames_from_suite(suite)
    ['file1.py', 'file1.rst', 'notatest1.foo']
    

    Here the .rst file were registered as unit doctest, while the .foo file was registered as functional doctest.

    To collect only unit doctests with a different set of filename extensions you can use uextensions:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     uextensions=['.foo'])
    >>> suite = test_suite()
    >>> get_basenames_from_suite(suite)
    ['file1.py', 'file1.txt', 'notatest1.foo', 'subdirfile.txt']
    

    Here the .foo file was registered as unit doctest and the .txt files as functional ones.

  • encoding:

    the encoding of testfiles. ‘utf-8’ by default. Setting this to None means using the default value. We’ve hidden one doctest file, that contains umlauts. If we set the encoding to ascii, we get an error:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     encoding='ascii')
    >>> suite = test_suite()
    Traceback (most recent call last):
    ...
    UnicodeDecodeError: 'ascii' codec can't decode ...: ordinal
    not in range(128)
    

    While using ‘latin-1’ will work:

    >>> test_suite = z3c.testsetup.register_all_tests(
    ...     'z3c.testsetup.tests.cave',
    ...     encoding='latin-1')
    >>> suite = test_suite()
    

    No traceback here.

    You can always overwrite an encoding setting for a certain file by following PEP 0263 ( http://www.python.org/dev/peps/pep-0263/ ).

  • checker:

    An output checker for functional doctests. None by default. A typical output checker can be created like this:

    >>> import re
    >>> from zope.testing import renormalizing
    >>> mychecker = renormalizing.RENormalizing([
    ...    (re.compile('[0-9]*[.][0-9]* seconds'),
    ...     '<SOME NUMBER OF> seconds'),
    ...    (re.compile('at 0x[0-9a-f]+'), 'at <SOME ADDRESS>'),
    ... ])
    

    This would match for example output like 0.123 seconds if you write in your doctest:

    <SOME NUBMER OF> seconds
    

    Please see testrunner.txt for examples of usage.

    Checkers are applied to functional doctests only!

  • globs:

    A dictionary of things that should be available immediately (without imports) during tests. Defaults are:

    dict(http=HTTPCaller(),
         getRootFolder=getRootFolder,
         sync=sync)
    

    for functional doctests and an empty dict for unit doctests. Python test globals can’t be set this way.

    If you want to register special globals for functional doctest or unit doctests only, then you can use the fglobs and/or uglobs keyword respectively. These keywords replace any globs value for the respective kind of tests.

    For more extensive examples see testrunner.txt.

  • setup:

    A function that takes a test argument and is executed before every single doctest. By default it runs:

    zope.app.testing.functional.FunctionalTestSetup().setUp()
    

    for functional doctests and an empty function for unit doctests. Python tests provide their own setups.

    If you want to register special setup-functions for either functional or unit doctests, then you can pass keyword parameters fsetup or usetup respectively.

  • teardown:

    The equivalent to setup. Runs by default:

    FunctionalTestSetup().tearDown()
    

    for functional doctests and:

    zope.testing.cleanup.cleanUp()
    

    for unit doctests. Python tests have to provide their own teardown functions in TestCases.

  • optionflags:

    Optionflags influence the behaviour of the testrunner. They are logically or’d so that you can add them arithmetically. See

    http://svn.zope.org/zope.testing/trunk/src/zope/testing/doctest.py

    for details.

  • zcml_config:

    A filepath of a ZCML file which is registered with functional doctests. In the ZCML file you can for example register principals (users) usable by functional doctests.

    By default an empty ZCML file of the z3c.testsetup package is used (ftesting.zcml).

    This parameter has no effect, if also a layer parameter is given.

  • layer_name:

    You can name your layer, to distinguish different setups of functional doctests. The layer name can be an arbitrary string.

    This parameter has no effect, if also a layer parameter is given.

  • layer:

    You can register a ZCML layer yourself and pass it as the layer parameter. If you only have a filepath to the according ZCML file, use the zcml_config paramter instead.

    This parameter overrides any zcml_config and layer_name parameter.

How to mark testfiles/modules

To avoid non-wanted files and modules to be registered, you have to mark your wanted test files/modules with a special string explicitly:

  • python modules you want to register must provide a module docstring that contains a line:

    :Test-Layer: python
    

    A module doctring is written at the top of file like this:

    Python Unit Test Example::

    """
    A module that tests things.
    
    :Test-Layer: python
    
    """
    import unittest
    class MyTest(unittest.TestCase):
        def testFoo(self):
            pass
    
  • doctest files that contain unit tests must provide a string:

    :Test-Layer: unit
    

    to be registered. Futhermore, their filename extension must be by default ‘.txt’ or ‘.rst’. A file sampletest.txt with a unit doctest therefore might look like this:

    Unit Doctest Example 1::

    ==========
    My package
    ==========
    
    :Test-Layer: unit
    
    This is documentation for the MyPackage package.
    
       >>> 1+1
       2
    

    Also python modules which contain tests in doctests notation are doctests. As rule of thumb you can say: if a module contains tests that are written preceeded by ‘>>>’, then this is a doctest. If unittest.TestCase classes are defined, then it is a ‘normal’ python testfile. Another valid unit doctest module therefore can look like this:

    Unit Doctest Example 2::

    """
    ==========
    My package
    ==========
    
    A package for doing things.
    
    :Test-Layer: unit
    
    We check for basic things::
    
       >>> 1+1
       2
    
    """
    class MyClass:
        pass
    
  • files that contain functional doctests must provide a string:

    :Test-Layer: functional
    

    to be registered. Furthermore they must by default have a filename extension .txt or .rst. A file sampletest.txt with functional tests might look like this:

    Functional Doctest Example::

    ==========
    My package
    ==========
    
    :Test-Layer: functional
    
    This is documentation for the MyPackage package.
    
       >>> 1+1
       2
    

CHANGES

0.1.0 (2008-02-15)

  • Initial Release

Download

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
z3c.testsetup-0.1.0-py2.4.egg (78.1 kB) Copy SHA256 hash SHA256 Egg 2.4
z3c.testsetup-0.1.0.tar.gz (47.6 kB) Copy SHA256 hash SHA256 Source None
z3c.testsetup-0.1.0.zip (72.6 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page