z3c.coverage 2.0.1

A script to visualize coverage reports via HTML

This package contains tools to work with Python coverage data.

coveragereport produces HTML reports from coverage data, with syntax-highlighted source code and per-package aggregate numbers.

coveragediff compares two sets of coverage reports and reports regressions, that is, increases in the number of untested lines of code.

Contents

• Using coveragereport

• Using coveragediff

• Getting coverage data

  • zope.testrunner

  • coverage.py

  • trace.py

• Frequently Asked Questions

  • Why use z3c.coverage instead of coverage html?

  • Should I use zope.testrunner’s built-in coverage, or coverage run bin/test?

  • Did anyone actually ask any of these questions?

• Changes

  • 2.0.1 (2013-10-01)

  • 2.0.0 (2013-02-20)

  • 1.3.1 (2012-10-24)

  • 1.3.0 (2012-09-06)

  • 1.2.0 (2010-02-11)

  • 1.1.3 (2009-07-24)

  • 1.1.2 (2008-04-14)

  • 1.1.1 (2008-01-31)

  • 1.1.0 (2008-01-29)

  • 1.0.1 (2007-09-26)

  • 1.0.0 (2007-09-26)

  • 0.2.1

  • 0.2.0

  • 0.1.0

Using coveragereport

$ coveragereport --help
Usage: coveragereport [options] [inputpath [outputdir]]

Converts coverage reports to HTML.  If the input path is omitted, it defaults
to coverage or .coverage, whichever exists.  If the output directory is
omitted, it defaults to inputpath + /report or ./coverage-reports, depending
on whether the input path points to a directory or a file.

Options:
  -h, --help            show this help message and exit
  -q, --quiet           be quiet
  -v, --verbose         be verbose (default)
  --strip-prefix=PREFIX
                        strip base directory from filenames loaded from
                        .coverage
  --path-alias=PATH=LOCALPATH
                        define path mappings for filenames loaded from
                        .coverage

Example use with zope.testrunner:

$ bin/test --coverage=coverage
$ coveragereport
$ ln -s mypackage.html coverage/report/index.html
$ xdg-open coverage/report/index.html
$ xdg-open coverage/report/all.html

Example use with nose and coverage.py:

$ nosetests --with-coverage --cover-erase
$ coveragereport --strip-prefix=/full/path/to/source/
$ ln -s mypackage.html coverage-reports/index.html
$ xdg-open coverage-reports/index.html
$ xdg-open coverage-reports/all.html

Sample report:

Note

You need enscript installed and available in your $PATH if you want syntax highlighting.

Using coveragediff

Usage: coveragediff [options] olddir newdir

Options:
  -h, --help         show this help message and exit
  --include=REGEX    only consider files matching REGEX
  --exclude=REGEX    ignore files matching REGEX
  --email=ADDR       send the report to a given email address (only if
                     regressions were found)
  --from=ADDR        set the email sender address
  --subject=SUBJECT  set the email subject
  --web-url=BASEURL  include hyperlinks to HTML-ized coverage reports at a
                     given URL

Usage example with zope.testrunner:

$ bin/test --coverage=coverage
$ vi src/...
$ mv coverage coverage.old
$ bin/test --coverage=coverage
$ coveragediff coverage.old coverage

You cannot use coveragediff with coverage.py data. More on that below.

Output example:

$ coveragediff coverage.old coverage
my.package.module: 36 new lines of untested code
my.package.newmodule: new file with 15 lines of untested code (out of 23)

Output with clickable links:

$ coveragediff coverage.old coverage --web-url=http://example.com/coverage
my.package.module: 36 new lines of untested code
See http://example.com/coverage/my.package.module.html

my.package.newmodule: new file with 15 lines of untested code (out of 23)
See http://example.com/coverage/my.package.newmodule.html

Output via email, convenient for continuous integration:

$ coveragediff coverage.old coverage --web-url=http://example.com/coverage \
                   --email 'Developers <[e-mail]>' \
                   --from 'Buildbot <[e-mail]>'

That last example doesn’t produce any output, but sends an email (via SMTP to localhost:25).

Getting coverage data

zope.testrunner

zope.testrunner can produce a directory full of files named dotted.package.name.cover that contain source code annotated with coverage information. To get them, use

bin/test --coverage=outdir/

Both coveragereport and coveragediff accept this as inputs.

coverage.py

coverage.py can produce a .coverage file, which is actually a Python pickle containing (incomplete) coverage information. To get it, use

coverage run bin/testrunner

coveragereport can take the .coverage file as an input, but it also needs access to the matching source files. And you have to manually specify the absolute pathname prefix of your source tree so that the report know how to translate filenames into dotted package names. Also, it’s not enough to have absolute pathnames, you need to supply the canonical absolute pathname (with no symlink segments), such as returned by os.path.realpath. This is very inconvenient. Sorry.

coveragediff is unable to compare two .coverage files and report regressions. One reason for that is the incompleteness of the data format (the pickle contains line numbers of executed statements, but doesn’t say which lines contain code and which ones are blank/comments/continuation lines/excluded source lines). The other reason is simpler: nobody wrote the code. ;)

Unfortunately coverage annotate does not produce files compatible with coveragereport/coveragediff. This could also be remedied if somebody wrote a patch.

Note

If you want to use a .coverage file produced on another machine or simply in a different working directory, you will need to tell coveragereport how to adjust the absolute filenames so that the sources can be found. Use the --path-alias option for that. Alternatively you could use coverage combine to manipulate the .coverage file itself, as described in the documentation.

Note

.coverage files are Python pickles, which has important security ramifications. You can craft a Pickle file that executes arbitrary code during load time. Do not ever attempt to use pickles received from untrusted sources.

trace.py

The *.cover annotated-source format produced by zope.testrunner actually comes from the Python standard library module trace.py. You can probably use trace.py directly. I’ve never tried.

Frequently Asked Questions

Why use z3c.coverage instead of coverage html?

Some people prefer the look of the reports produced by z3c.coverage. Some people find per-package coverage summaries or the tree-like navigation convenient.

Should I use zope.testrunner’s built-in coverage, or coverage run bin/test?

coverage.py is much faster, but using it (and hooking it up to z3c.coverage) is perhaps less convenient. E.g. if you use zc.buildout 1.5.x with zc.recipe.testrunner, you will be unable to use coverage run bin/test because of mystic semi-broken site isolation magic of the former.

Did anyone actually ask any of these questions?

Does asking myself count?

Changes

2.0.1 (2013-10-01)

• Bug: coveragereport now also accepts non-ASCII chars in source files.

2.0.0 (2013-02-20)

• Added Python 3.3 and PyPy 1.9 support.

• Dropped Python 2.4 and 2.5 support.

1.3.1 (2012-10-24)

• Nicer PyPI description. Doctests are tests, not docs.

• coveragereport now accepts --path-alias.

• coveragereport: new color step between yellow (90%) and green (100%), a yellowish-green (95%).

1.3.0 (2012-09-06)

• coveragereport now accepts --help, --verbose and --quiet options, with verbose being on by default.

• coveragereport now can handle .coverage files produced by http://pypi.python.org/pypi/coverage

• Bugfix: sorting by numbered of uncovered lines was broken in the all.html report.

1.2.0 (2010-02-11)

• Rename the coverage script to coveragereport, to avoid name clashes with Ned Batchelder’s excellent coverage.py.

1.1.3 (2009-07-24)

• Bug: Doctest did not normalize the whitespace in coveragediff.txt. For some reason it passes while testing independently, but when running all tests, it failed.

1.1.2 (2008-04-14)

• Bug: When a package path contained anywhere the word “test”, it was ignored from the coverage report. The intended behavior, however, was to ignore files that relate to setting up tests.

• Bug: Sort the results of os.listdir() in README.txt to avoid non-deterministic failures.

• Bug: The logic for ignoring unit and functional test modules also used to ignore modules and packages called testing.

• Change “Unit test coverage” to “Test coverage” in the title – it works perfectly fine for functional tests too.

1.1.1 (2008-01-31)

• Bug: When the package was released, the test which tests the availability of an SVN revision number failed. Made the test more reliable.

1.1.0 (2008-01-29)

• Feature: The main() coverage report function now accepts the arguments of the script as a function argument, making it easier to configure the script from buildout.

• Feature: When the report directory does not exist, the report generator creates it for you.

• Feature: Eat your own dog food by creating a buildout that can create coverage reports.

• Bug: Improved the test coverage to 100%.

1.0.1 (2007-09-26)

• Bug: Fixed meta-data.

1.0.0 (2007-09-26)

• First public release.

0.2.1

• Feature: Added the --web option to coveragediff.

• Feature: Added a test suite.

0.2.0

• Feature: Added coveragediff.py.

0.1.0

• Initial release of coveragereport.py.