z3c.recipe.compattest 2.0

Buildout recipe to create testrunners for testing compatibility with other packages

z3c.recipe.compattest

This buildout recipe generates a list of packages to test and a test runner that runs each package’s tests (isolated from any other tests). This is useful to check that the changes made while developing a package do not break any packages that are using this package.

Usage

Add a part to your buildout.cfg that uses this recipe. No further configuration is required, but you can set the following options:

• include: list of packages to include (whitespace-separated) (default: empty)

• include-dependencies: list of packages to include including their direct dependencies. (default: empty)

• exclude: packages matching any regex in this list will be excluded (default: empty)

• script: the name of the runner script (defaults to the part name)

>>> cd(sample_buildout)
>>> write('buildout.cfg', """
... [buildout]
... parts = compattest
...
... [compattest]
... recipe = z3c.recipe.compattest
... include = z3c.recipe.compattest
... """)

>>> 'Installing compattest' in system(buildout)
True

Details

The recipe generates a test runner for each package, as well as a global runner script (called test-compat by default) that will run all of them:

>>> ls('bin')
- buildout
- compattest
- compattest-z3c.recipe.compattest

>>> cat('bin', 'compattest')
#!...py...
...main(...compattest-z3c.recipe.compattest...

We take care about installing the test dependencies for the packages (from their extras_require['test']). To demonstrate this, we declared a (superfluous) test dependency on zope.dottedname, which is picked up (if the package is already installed in a virtual environment, we cannot find these dependencies):

>>> try:
...     print('start')
...     cat('parts', 'compattest-z3c.recipe.compattest', 'site-packages', 'site.py')
... except IOError:
...     # When the tests are run from a virtualenv, the bin scripts are created
...     # in a different location, and if we are also installed in
...     # that location, we don't have to install any extras ourself.
...     cat('bin', 'compattest-z3c.recipe.compattest')
...     print('zope.dottedname')
start
...zope.dottedname...

If we use include-dependencies instead of just include, its direct dependencies are also picked up, for instance zc.buildout:

>>> write('buildout.cfg', """
... [buildout]
... parts = compattest
...
... [compattest]
... recipe = z3c.recipe.compattest
... include-dependencies = z3c.recipe.compattest
... """)
>>> print('start' + system(buildout))
start...
...
Generated script '/sample-buildout/bin/compattest-zc.buildout'.
...

All our direct dependencies have a test script now:

>>> ls('bin')
- buildout
- compattest
- compattest-z3c.recipe.compattest
- compattest-zc.buildout
- compattest-zc.recipe.testrunner

And if you want to exclude one of the automatically included dependencies, use the exclude option:

>>> write('buildout.cfg', """
... [buildout]
... parts = compattest
...
... [compattest]
... recipe = z3c.recipe.compattest
... include-dependencies = z3c.recipe.compattest
... exclude = zc.buildout
... """)
>>> print('start' + system(buildout))
start...
Generated script '/sample-buildout/bin/compattest'...

bin/compattest-zc.buildout is now missing:

>>> ls('bin')
- buildout
- compattest
- compattest-z3c.recipe.compattest
- compattest-zc.recipe.testrunner

Passing options to the test runners

If you want to use custom options in the generated test runners, you can specify them in the part options, prefixed by runner-. That is, if you want to pass the --foo option by default to all generated test runners, you can set runner-defaults = ['--foo'] in your part:

>>> write('buildout.cfg', """
... [buildout]
... parts = compattest
...
... [compattest]
... recipe = z3c.recipe.compattest
... include = z3c.recipe.compattest
... runner-defaults = ['-c', '-v', '-v']
... """)
>>> ignore = system(buildout)
>>> cat('bin', 'compattest-z3c.recipe.compattest')
#!...py...
...run(...['-c', '-v', '-v']...

Every options prefixed by runner- will be automatically passed to the generated test runners.

Passing Extra paths to the test runners

If you want to add some paths to the generated test runners, you can do it with the extra-paths option in the part. This might be interesting if you want to test packages that depends on zope2 < 2.12:

>>> write('buildout.cfg', """
... [buildout]
... parts = compattest
...
... [compattest]
... recipe = z3c.recipe.compattest
... include = z3c.recipe.compattest
... extra-paths = zope2location/lib/python
... """)
>>> ignore = system(buildout)
>>> try:
...     print('start')
...     cat('parts', 'compattest-z3c.recipe.compattest', 'site-packages', 'site.py')
... except IOError:
...     print('start')
...     # When the tests are run from a virtualenv, the bin scripts are created
...     # in a different location.
...     cat('bin', 'compattest-z3c.recipe.compattest')
start
...zope2location/lib/python...

CHANGES

2.0 (2023-02-20)

• Add support for Python 3.9, 3.10, 3.11.

• Drop support for Python 2.7, 3.5, 3.6.

1.1.0 (2020-05-14)

• Drop support for Python 2.6, 3.2, 3.3 and 3.4.

• Add support for Python 3.5, 3.6, 3.7, 3.8, PyPy2 and PyPy3.

• Fix file descriptor leaks. See issue 1.

1.0 (2013-03-02)

• Depend on buildout 2 and zc.recipe.testrunner 2.

0.13.1 (2010-12-17)

• Fix tests on windows.

• Fix for use with a python executable from inside a virtualenv.

0.13 (2010-10-07)

• Depend on and use the new features of the zc.buildout 1.5 line. At the same time support for zc.buildout <= 1.5.1 has been dropped.

• Updated test set up, to run with newer zope.testing version which no longer includes testrunner.

• The z3c.recipe.scripts.scripts recipe behind zc.recipe.testrunner.TestRunner does not accept plain dicts anymore, so we wrap the options in a _BackwardsSupportOptions object. Ideally this should’ve use an official API though.

0.12.2 (2010-02-24)

• Moved the gathering of include-dependencies from the __init__ to the update method to prevent installing dependencies before other buildout parts could do their job.

0.12.1 (2009-12-15)

• Fixed bug in using exclude introduced in 0.12 (including test to make sure it doesn’t happen again).

0.12 (2009-12-14)

• Added include-dependencies option that automatically includes the dependencies of the specified packages. Very handy to get an automatically updated list of those packages that are most useful to test: all our dependencies.

0.11 (2009-09-30)

• Removed the “check out packages from subversion” feature. If you require such functionality, mr.developer <http://pypi.python.org/pypi/mr.developer> provides this much more comprehensively (and for multiple version control systems, too) .

0.10 (2009-09-28)

• Options prefixed by runner- are automatically passed to generated test runners.

0.9 (2009-09-14)

• Test runner: return the exit code 1 in case of test failures; this simplifies buildbot configurations.

0.8 (2009-08-17)

• Windoes is now supported.

• Changed the default master script name to the part name. (Don’t add a “test-” prefix any more.)

0.7 (2009-08-13)

• Simplified building the list of packages even more: we now just take a list of packages, period.

0.6 (2009-08-07)

• Restructured the way we construct our list of packages to test: We no longer filter a list we retrieved from SVN with includes/excludes, but use an explicit list that can be populated from a buildout section, e. g. [versions]. Thus, we can now easily test against a KGS.

• Always enable all extras of packages under test.

0.5 (2009-01-29)

• Fix duplicate url parameter in setup.py that confused Python 2.4 but got accepted by Python 2.5.

0.4 (2009-01-29)

• Ignore missing package releases for packages listed in Subversion (as long as we don’t try to run from Subversion).

• Allow parallel execution of the individual test runners by stating ‘max_jobs=X’ in the recipe’s options.

0.3 (2009-01-28)

• Adding the exclude parameter in buildout causes the default exclude list to be merged with the option in buildout.cfg.

0.2 (2009-01-28)

• Implemented use_svn option to use SVN trunk checkouts instead of released versions.

0.1 (2009-01-28)

• first released version