zope.testrunner 6.6

Zope testrunner script.

zope.testrunner

Contents

• zope.testrunner

• Using zope.testrunner

  • Installation

    • Buildout-based projects

    • Virtualenv-based projects

  • Some useful command-line options to get you started

  • Writing tests

  • Test grouping

  • Other features

• zope.testrunner Changelog

  • 6.6 (2024-10-16)

  • 6.5 (2024-08-06)

  • 6.4 (2024-02-27)

  • 6.3.1 (2024-02-12)

  • 6.3 (2024-02-07)

  • 6.2.1 (2023-12-22)

  • 6.2 (2023-11-08)

  • 6.1 (2023-08-26)

  • 6.0 (2023-03-28)

  • 5.6 (2022-12-09)

  • 5.5.1 (2022-09-07)

  • 5.5 (2022-06-24)

  • 5.4.0 (2021-11-19)

  • 5.3.0 (2021-03-17)

  • 5.2 (2020-06-29)

  • 5.1 (2019-10-19)

  • 5.0 (2019-03-19)

  • 4.9.2 (2018-11-24)

  • 4.9.1 (2018-11-21)

  • 4.9 (2018-10-05)

  • 4.8.1 (2017-11-12)

  • 4.8.0 (2017-11-10)

  • 4.7.0 (2017-05-30)

  • 4.6.0 (2016-12-28)

  • 4.5.1 (2016-06-20)

  • 4.5.0 (2016-05-02)

  • 4.4.10 (2015-11-10)

  • 4.4.9 (2015-05-21)

  • 4.4.8 (2015-05-01)

  • 4.4.7 (2015-04-02)

  • 4.4.6 (2015-01-21)

  • 4.4.5 (2015-01-06)

  • 4.4.4 (2014-12-27)

  • 4.4.3 (2014-03-19)

  • 4.4.2 (2014-02-22)

  • 4.4.1 (2013-07-10)

  • 4.4.0 (2013-06-06)

  • 4.3.3 (2013-03-03)

  • 4.3.2 (2013-03-03)

  • 4.3.1 (2013-03-02)

  • 4.3.0 (2013-03-02)

  • 4.2.0 (2013-02-12)

  • 4.1.1 (2013-02-08)

  • 4.1.0 (2013-02-07)

  • 4.0.4 (2011-10-25)

  • 4.0.3 (2011-03-17)

  • 4.0.2 (2011-03-16)

  • 4.0.1 (2011-02-21)

  • 4.0.0 (2010-10-19)

  • 4.0.0b5 (2010-07-20)

  • 4.0.0b4 (2010-06-23)

  • 4.0.0b3 (2010-06-16)

  • 4.0.0b2 (2010-05-03)

  • 4.0.0b1 (2010-04-29)

This package provides a flexible test runner with layer support.

Detailed documentation is hosted at https://zopetestrunner.readthedocs.io

Using zope.testrunner

Installation

Buildout-based projects

zope.testrunner is often used for projects that use buildout:

[buildout]
develop = .
parts = ... test ...

[test]
recipe = zc.recipe.testrunner
eggs = mypackage

The usual buildout process

python bootstrap.py
bin/buildout

creates a bin/test script that will run the tests for mypackage.

Tip

zc.recipe.testrunner takes care to specify the right --test-path option in the generated script. You can add other options (such as --tests-pattern) too; check zc.recipe.testrunner’s documentation for details.

Virtualenv-based projects

pip install zope.testrunner and you’ll get a zope-testrunner script. Run your tests with

zope-testrunner --test-path=path/to/your/source/tree

Your source code needs to be available for the testrunner to import, so you need to run python setup.py install or pip install -e . into the same virtualenv.

Some useful command-line options to get you started

-p

show a progress indicator

-v

increase verbosity

-c

colorize the output

-t test

specify test names (one or more regexes)

-m module

specify test modules (one or more regexes)

-s package

specify test packages (one or more regexes)

--list-tests

show names of tests instead of running them

-x

stop on first error or failure

-D, --pdb

enable post-mortem debugging of test failures

--xml path

generate XML reports to be written at the given path

--help

show all command-line options (there are many more!)

For example

bin/test -pvc -m test_foo -t TestBar

runs all TestBar tests from a module called test_foo.py.

Writing tests

zope.testrunner expects to find your tests inside your package directory, in a subpackage or module named tests. Test modules in a test subpackage should be named test*.py.

Tip

You can change these assumptions with --tests-pattern and --test-file-pattern test runner options.

Tests themselves should be classes inheriting from unittest.TestCase, and if you wish to use doctests, please tell the test runner where to find them and what options to use for them in by supplying a function named test_suite.

Example:

import unittest
import doctest

class TestArithmetic(unittest.TestCase):

    def test_two_plus_two(self):
        self.assertEqual(2 + 2, 4)


def doctest_string_formatting():
    """Test Python string formatting

        >>> print('{} + {}'.format(2, 2))
        2 + 2

    """

def test_suite():
    return unittest.TestSuite([
        unittest.defaultTestLoader.loadTestsFromName(__name__),
        doctest.DocTestSuite(),
        doctest.DocFileSuite('../README.txt',
                             optionflags=doctest.ELLIPSIS),
    ])

Test grouping

In addition to per-package and per-module filtering, zope.testrunner has other mechanisms for grouping tests:

• layers allow you to have shared setup/teardown code to be used by a group of tests, that is executed only once, and not for each test. Layers are orthogonal to the usual package/module structure and are specified by setting the layer attribute on test suites.

• levels allow you to group slow-running tests and not run them by default. They’re specified by setting the level attribute on test suites to an int.

Other features

zope.testrunner can profile your tests, measure test coverage, check for memory leaks, integrate with subunit, shuffle the test execution order, and run multiple tests in parallel.

zope.testrunner Changelog

6.6 (2024-10-16)

• Make tests compatible with Python 3.13 + add support for that version. (#181)

• Drop support for Python 3.7.

6.5 (2024-08-06)

• Remove setuptools fossils.

• unittest.TestCase.subTest support (#91).

• remove support for setup.py’s test command. Support for this command has been dropped by modern setuptools versions and correspondingly has been removed from most zopefoundation packages; zope.testrunner now follows.

• setup.py’s ftest command is now only supported when the used setuptools version still supports test.

6.4 (2024-02-27)

• Add PEP 420 support (implicit namespaces). (#160)

6.3.1 (2024-02-12)

• Fix XML tests when running in distribution resp. separately. (#163)

6.3 (2024-02-07)

• Exit cleanly when using the test runner --version argument. (#102)

• Add new --xml <path> option to write JUnit-like XML reports. Code comes from collective.xmltestreport, but be aware that here --xml is not a boolean, but expects a path! (#148).

• Add support for Python 3.13a3.

6.2.1 (2023-12-22)

• Work around Python 3.12.1+ no longer calling startTest for skipped tests (#157).

6.2 (2023-11-08)

• Add support for Python 3.12.

• Update code and tests to python-subunit >= 1.4.3 thus requiring at least this version.

6.1 (2023-08-26)

• Add preliminary support for Python 3.12b4. (#149)

6.0 (2023-03-28)

• Drop support for Python 2.7, 3.5, 3.6.

5.6 (2022-12-09)

• Add support for Python 3.11.

• Inline a small part of random.Random.shuffle which was deprecated in Python 3.9 and removed in 3.11 (#119).

• Don’t trigger post mortem debugger for skipped tests. ( #141).

5.5.1 (2022-09-07)

• Fix: let --at-level=level with level <= 0 run the tests at all levels (rather than at no level) #138.

5.5 (2022-06-24)

• Use sys._current_frames (rather than threading.enumerate) as base for new thread detection, fixes #130.

• New option --gc-after-test. It calls for a garbage collection after each test and can be used to track down ResourceWarning``s and cyclic garbage. With ``rv = gc.collect(), ! is output on verbosity level 1 when rv is non zero (i.e. when cyclic structures have been released), [``*rv*]`` on higher verbosity levels and a detailed cyclic garbage analysis on verbosity level 4+. For details, see #133.

• Allow the filename for the logging configuration to be specified via the envvar ZOPE_TESTRUNNER_LOG_INI. If not defined, the configuration continues to be locked for in file log.ini of the current working directory. Remember the logging configuration file in envvar ZOPE_TESTRUNNER_LOG_INI to allow spawned child processes to recreate the logging configuration. For details, see #134.

5.4.0 (2021-11-19)

• Improve --help documentation for --package-path option (#121).

• Do not disable existing loggers during logsupport initialization (#120).

• Fix tests with testtools >= 2.5.0 (#125).

• Add support for Python 3.10.

5.3.0 (2021-03-17)

• Add support for Python 3.9.

• Fix package init file missing warning (#112).

• Make standard streams provide a buffer attribute on Python 3 when using –buffer or testing under subunit.

5.2 (2020-06-29)

• Add support for Python 3.8.

• When a layer is run in a subprocess, read its stderr in a thread to avoid a deadlock if its stderr output (containing failing and erroring test IDs) overflows the capacity of a pipe (#105).

5.1 (2019-10-19)

• Recover more gracefully when layer setUp or tearDown fails, producing useful subunit output.

• Prevent a spurious warning from the --require-unique option if the --module option was not used.

• Add optional buffering of standard output and standard error during tests, requested via the --buffer option or enabled by default for subunit.

• Fix incorrect failure counts in per-layer summary output, broken in 4.0.1.

5.0 (2019-03-19)

• Fix test failures and deprecation warnings occurring when using Python 3.8a1. (#89)

• Drop support for Python 3.4.

4.9.2 (2018-11-24)

• Fix TypeError: a bytes-like object is required, not 'str' running tests in parallel on Python 3. See issue 80.

4.9.1 (2018-11-21)

• Fix AssertionError in _DummyThread.isAlive on Python 3 (#81).

4.9 (2018-10-05)

• Drop support for Python 3.3.

• Add support for Python 3.7.

• Enable test coverage reporting on coveralls.io and in tox.ini.

• Host documentation at https://zopetestrunner.readthedocs.io

• Remove untested support for the --pychecker option. See issue 63.

• Update the command line interface to use argparse instead of optparse. See issue 61.

• Use ipdb instead of pdb for post-mortem debugging if available (#10).

• Add a –require-unique option to check for duplicate test IDs. See LP #682771.

• Reintroduce optional support for subunit, now with support for both version 1 and version 2 of its protocol.

• Handle string in exception values when formatting chained exceptions. (#74)

4.8.1 (2017-11-12)

• Enable DeprecationWarning earlier, when discovering test modules. This lets warnings that are raised on import (such as those produced by zope.deprecation.moved) be reported. See issue 57.

4.8.0 (2017-11-10)

• Automatically enable DeprecationWarning when running tests. This is recommended by the Python core developers and matches the behaviour of the unittest module. This can be overridden with Python command-line options (-W) or environment variables (PYTHONWARNINGS). See issue 54.

4.7.0 (2017-05-30)

• Drop all support for subunit.

4.6.0 (2016-12-28)

• Make the subunit support purely optional: applications which have been getting the dependencies via zope.testrunner should either add zope.testrunner[subunit] to their install_requires or else depend directly on python-subunit.

• New option --ignore-new-thread=<regexp> to suppress “New thread(s)” warnings.

• Support Python 3.6.

4.5.1 (2016-06-20)

• Fixed: Using the -j option to run tests in multiple processes caused tests that used the multiprocessing package to hang (because the testrunner replaced sys.stdin with an unclosable object).

• Drop conditional dependency on unittest2 (redundant after dropping support for Python 2.6).

4.5.0 (2016-05-02)

• Stop tests for all layers when test fails/errors when started with -x/–stop-on-error (#37).

• Drop support for Python 2.6 and 3.2.

4.4.10 (2015-11-10)

• Add support for Python 3.5 (#31).

• Insert extra paths (from --path) to the front of sys.argv (#32).

4.4.9 (2015-05-21)

• When using -j, parallelize all the tests, including the first test layer (#28).

4.4.8 (2015-05-01)

• Support skipped tests in subunit output (#25).

• More efficient test filtering (#26).

4.4.7 (2015-04-02)

• Work around a bug in PyPy3’s curses module (#24).

4.4.6 (2015-01-21)

• Restore support for instance-based test layers that regressed in 4.4.5 (#20).

4.4.5 (2015-01-06)

• Sort related layers close to each other to reduce the number of unnecessary teardowns (fixes #14).

• Run the unit test layer first (fixes LP #497871).

4.4.4 (2014-12-27)

• When looking for the right location of test code, start with longest location paths first. This fixes problems with nested code locations.

4.4.3 (2014-03-19)

• Added support for Python 3.4.

4.4.2 (2014-02-22)

• Drop support for Python 3.1.

• Fix post-mortem debugging when a non-printable exception happens (https://github.com/zopefoundation/zope.testrunner/issues/8).

4.4.1 (2013-07-10)

• Updated boostrap.py to version 2.2.

• Fix nondeterministic test failures on Python 3.3

• Tear down layers after post_mortem debugging is finished.

• Fix tests that write to source directory, it might be read-only.

4.4.0 (2013-06-06)

• Fix tests selection when the negative “!” pattern is used several times (LP #1160965)

• Moved tests into a ‘tests’ subpackage.

• Made python -m zope.testrunner work again.

• Support ‘skip’ feature of unittest2 (which became the new unittest in Python 2.7).

• Better diagnostics when communication with subprocess fails (https://github.com/zopefoundation/zope.testrunner/issues/5).

• Do not break subprocess execution when the test suite changes the working directory (https://github.com/zopefoundation/zope.testrunner/issues/6).

• Count test module import errors as errors (LP #1026576).

4.3.3 (2013-03-03)

• Running layers in sub-processes did not use to work when run via python setup.py ftest since it tried to run setup.py with all the command line options. It now detects setup.py runs and we run the test runner directly.

4.3.2 (2013-03-03)

• Fix SkipLayers class in cases where the distribution specifies a test_suite value.

4.3.1 (2013-03-02)

• Fixed a bug in the ftest command and added a test.

• Fixed a trivial test failure with Python 3 of the previous release.

4.3.0 (2013-03-02)

• Expose ftest distutils command via an entry point.

• Added tests for zope.testrunner.eggsupport.

4.2.0 (2013-02-12)

• Dropped use of 2to3, rewrote source code to be compatible with all Python versions. Introduced a dependency on six.

4.1.1 (2013-02-08)

• Dropped use of zope.fixers (LP: #1118877).

• Fixed tox test error reporting; fixed tests on Pythons 2.6, 3.1, 3.2, 3.3 and PyPy 1.9.

• Fix –shuffle ordering on Python 3.2 to be the same as it was on older Python versions.

• Fix –shuffle nondeterminism when multiple test layers are present. Note: this will likely change the order of tests for the same –shuffle-seed.

• New option: –profile-directory. Use it in the test suite so that tests executed by detox in parallel don’t conflict.

• Use a temporary coverage directory in the test suite so that tests executed by detox in parallel don’t conflict.

• Fix –post-mortem (aka -D, –pdb) when a test module cannot be imported or is invalid (LP #1119363).

4.1.0 (2013-02-07)

• Replaced deprecated zope.interface.implements usage with equivalent zope.interface.implementer decorator.

• Dropped support for Python 2.4 and 2.5.

• Made StartUpFailure compatible with unittest.TextTestRunner() (LP #1118344).

4.0.4 (2011-10-25)

• Work around sporadic timing-related issues in the subprocess buffering tests. Thanks to Jonathan Ballet for the patch!

4.0.3 (2011-03-17)

• Added back support for Python <= 2.6 which was broken in 4.0.2.

4.0.2 (2011-03-16)

• Added back Python 3 support which was broken in 4.0.1.

• Fixed Unexpected success support by implementing the whole concept.

• Added support for the new __pycache__ directories in Python 3.2.

4.0.1 (2011-02-21)

• LP #719369: An Unexpected success (concept introduced in Python 2.7) is no longer handled as success but as failure. This is a workaround. The whole unexpected success concept might be implemented later.

4.0.0 (2010-10-19)

• Show more information about layers whose setup fails (LP #638153).

4.0.0b5 (2010-07-20)

• Update fix for LP #221151 to a spelling compatible with Python 2.4.

• Timestamps are now always included in subunit output (r114849).

• LP #591309: fix a crash when subunit reports test failures containing UTF8-encoded data.

4.0.0b4 (2010-06-23)

• Package as a zipfile to work around Python 2.4 distutils bug (no feature changes or bugfixes in zope.testrunner itself).

4.0.0b3 (2010-06-16)

• LP #221151: keep unittest.TestCase.shortDescription happy by supplying a _testMethodDoc attribute.

• LP #595052: keep the distribution installable under Python 2.4: its distutils appears to munge the empty __init__.py file in the foo.bar egg used for testing into a directory.

• LP #580083: fix the bin/test script to run only tests from zope.testrunner.

• LP #579019: When layers were run in parallel, their tearDown was not called. Additionally, the first layer which was run in the main thread did not have its tearDown called either.

4.0.0b2 (2010-05-03)

• Having ‘sampletests’ in the MANIFEST.in gave warnings, but doesn’t actually seem to include any more files, so I removed it.

• Moved zope.testing.exceptions to zope.testrunner.exceptions. Now zope.testrunner no longer requires zope.testing except for when running its own tests.

4.0.0b1 (2010-04-29)

• Initial release of the testrunner from zope.testrunner as its own module. (Previously it was part of zope.testing.)