z3c.coverage 1.1.3

A script to visualize coverage reports via HTML

This package produces a nice HTML representation of the coverage data generated by the Zope test runner.

It also has a script to check for differences in coverage and report any regressions (increases in the number of untested lines).

Detailed Documentation

Test Coverage Reports

The main objective of this package is to convert the text-based coverage output into an HTML coverage report. This is simply done by specifying the directory of the test reports and the desired output directory.

Luckily we already have the text input ready:

>>> import os
>>> import z3c.coverage
>>> inputDir = os.path.join(
...     os.path.split(z3c.coverage.__file__)[0], 'sampleinput')

The output directory has to be created first:

>>> import tempfile
>>> outputDir = os.path.join(tempfile.mkdtemp(), 'report')

We can now simply create the coverage report as follows:

>>> from z3c.coverage import coveragereport
>>> coveragereport.main((inputDir, outputDir))

Looking at the output directory, we now see several files:

>>> print '\n'.join(sorted(os.listdir(outputDir)))
all.html
z3c.coverage.__init__.html
z3c.coverage.coveragediff.html
z3c.coverage.coveragereport.html
z3c.coverage.html
z3c.html

API Tests

CoverageNode Class

This class represents a node in the source tree. Simple modules are considered leaves and do not have children. Let’s create a node for the z3c namespace first:

>>> z3cNode = coveragereport.CoverageNode()

Before using the API, let’s create a few more nodes and a tree from it:

>>> coverageNode = coveragereport.CoverageNode()
>>> z3cNode['coverage'] = coverageNode

>>> reportNode = coveragereport.CoverageNode()
>>> reportNode._covered, reportNode._total = 40, 134
>>> coverageNode['coveragereport'] = reportNode

>>> diffNode = coveragereport.CoverageNode()
>>> diffNode._covered, diffNode._total = 128, 128
>>> coverageNode['coveragediff'] = diffNode

>>> initNode = coveragereport.CoverageNode()
>>> initNode._covered, initNode._total = 0, 0
>>> coverageNode['__init__'] = initNode

Let’s now have a look at the coverage of the z3c namespace:

>>> z3cNode.coverage
(168, 262)

We can also ask for the percentile:

>>> z3cNode.percent
64
>>> initNode.percent
100

We can ask for the amount of uncovered lines:

>>> z3cNode.uncovered
94

Finally, we also can get a nice output:

>>> print z3cNode
64% covered (94 of 262 lines uncovered)

index_to_filename() function

Takes an indexed Python path and produces the cover filename for it:

>>> coveragereport.index_to_filename(('z3c', 'coverage', 'coveragereport'))
'z3c.coverage.coveragereport.cover'

>>> coveragereport.index_to_filename(())
''

index_to_nice_name() function

Takes an indexed Python path and produces a nice “human-readable” string:

>>> coveragereport.index_to_nice_name(('z3c', 'coverage', 'coveragereport'))
'        coveragereport'

>>> coveragereport.index_to_nice_name(())
'Everything'

index_to_name() function

Takes an indexed Python path and produces a “human-readable” string:

>>> coveragereport.index_to_name(('z3c', 'coverage', 'coveragereport'))
'z3c.coverage.coveragereport'

>>> coveragereport.index_to_name(())
'everything'

percent_to_colour() function

Given a coverage percentage, this function returns a color to represent the coverage:

>>> coveragereport.percent_to_colour(100)
'green'
>>> coveragereport.percent_to_colour(92)
'yellow'
>>> coveragereport.percent_to_colour(85)
'orange'
>>> coveragereport.percent_to_colour(50)
'red'

get_svn_revision() function

Given a path, the function tries to determine the revision number of the file. If it fails, “UNKNOWN” is returned:

>>> path = os.path.split(z3c.coverage.__file__)[0]
>>> coveragereport.get_svn_revision(path) != 'UNKNOWN'
True

>>> coveragereport.get_svn_revision(path + '/__init__.py')
'UNKNOWN'

syntax_highlight() function

This function takes a cover file, converts it to a nicely colored HTML output:

>>> filename = os.path.join(
...     os.path.split(z3c.coverage.__file__)[0], '__init__.py')

>>> print coveragereport.syntax_highlight(filename)
<BLANKLINE>
<I><FONT COLOR="#B22222"># Make a package.
</FONT></I>

If the highlighing command is not available, no coloration is done:

>>> command_orig = coveragereport.HIGHLIGHT_COMMAND
>>> coveragereport.HIGHLIGHT_COMMAND = 'foobar %s'

>>> print coveragereport.syntax_highlight(filename)
# Make a package.
<BLANKLINE>

>>> coveragereport.HIGHLIGHT_COMMAND = command_orig

coveragereport.py is a script

For convenience you can download the coveragereport.py module and run it as a script:

>>> import sys
>>> sys.argv = ['coveragereport', inputDir, outputDir]

>>> script_file = os.path.join(
...     z3c.coverage.__path__[0], 'coveragereport.py')

>>> execfile(script_file, dict(__name__='__main__'))

Defaults are chosen, when no input and output dir is specified:

>>> def make_coverage_reports_stub(path, report_path):
...     print path
...     print report_path

>>> make_coverage_reports_orig = coveragereport.make_coverage_reports
>>> coveragereport.make_coverage_reports = make_coverage_reports_stub

>>> sys.argv = ['coveragereport']
>>> coveragereport.main()
coverage
coverage/reports

>>> coveragereport.make_coverage_reports = make_coverage_reports_orig

coveragediff internals

coveragediff is a tool that can be used to compare two directories with coverage reports (such as the ones produced by zope.testing test runner with the --coverage option, or, more generally, the trace module from the Python standard library). coveragediff reports regressions, that is, increases in the number of untested lines of code.

This document describes the internals of coveragediff.py. It also acts as a test suite.

Locating coverage files

The function find_coverage_files looks for plain-text coverage reports in a given directory

>>> import z3c.coverage, os
>>> sampleinput_dir = os.path.join(z3c.coverage.__path__[0], 'sampleinput')

>>> from z3c.coverage.coveragediff import find_coverage_files
>>> for filename in sorted(find_coverage_files(sampleinput_dir)):
...     print filename
z3c.coverage.__init__.cover
z3c.coverage.coveragediff.cover
z3c.coverage.coveragereport.cover
z3c.coverage.tests.cover

The function filter_coverage_files looks for plain-text coverage reports in a given location that match a set of include and exclude patterns

>>> from z3c.coverage.coveragediff import filter_coverage_files
>>> for filename in sorted(filter_coverage_files(sampleinput_dir)):
...     print filename
z3c.coverage.__init__.cover
z3c.coverage.coveragediff.cover
z3c.coverage.coveragereport.cover
z3c.coverage.tests.cover

>>> for filename in sorted(filter_coverage_files(sampleinput_dir,
...                                              include=['diff'])):
...     print filename
z3c.coverage.coveragediff.cover

The patterns are regular expressions

>>> for filename in sorted(filter_coverage_files(sampleinput_dir,
...                                              exclude=['^z'])):
...     print filename

Parsing coverage files

The function count_coverage reads a plain-text coverage reports and returns two numbers: the number of tested code lines and the number of untested code lines.

>>> from z3c.coverage.coveragediff import count_coverage
>>> filename = os.path.join(sampleinput_dir, 'z3c.coverage.tests.cover')
>>> tested, untested = count_coverage(filename)
>>> tested
10
>>> untested
3

Comparing coverage files

The function compare_file reads two coverage reports for the same module and reports a warning if the new file has more untested lines of code

>>> from z3c.coverage.coveragediff import compare_file
>>> another_dir = os.path.join(z3c.coverage.__path__[0], 'moresampleinput')
>>> old_filename = os.path.join(sampleinput_dir,
...                             'z3c.coverage.coveragediff.cover')
>>> new_filename = os.path.join(another_dir,
...                             'z3c.coverage.coveragediff.cover')
>>> compare_file(old_filename, new_filename)
z3c.coverage.coveragediff: 36 new lines of untested code

If the number of untested lines is the same or smaller than before, there’s no output

>>> compare_file(new_filename, new_filename)
>>> compare_file(new_filename, old_filename)

The function new_file is used to look for untested lines of code in new modules.

>>> from z3c.coverage.coveragediff import new_file
>>> new_filename = os.path.join(another_dir,
...                             'z3c.coverage.fakenewmodule.cover')
>>> new_file(new_filename)
z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)

Once again, if there are no untested lines, new_file is quiet

>>> new_filename = os.path.join(another_dir,
...                             'z3c.coverage.faketestedmodule.cover')
>>> new_file(new_filename)

Comparing directories

compare_dirs ties it all together: you pass in two directory names, you get a bunch of warnings about regressions

>>> from z3c.coverage.coveragediff import compare_dirs
>>> compare_dirs(sampleinput_dir, another_dir)
z3c.coverage.coveragediff: 36 new lines of untested code
z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)

You can pass include and exclude arguments as well

>>> compare_dirs(sampleinput_dir, another_dir, exclude=['[Ff]ake'])
z3c.coverage.coveragediff: 36 new lines of untested code

>>> compare_dirs(sampleinput_dir, another_dir, include=['d.ff'])
z3c.coverage.coveragediff: 36 new lines of untested code

MailSender

The MailSender class is responsible for assembling an RFC-2822 email message and handing it off to an SMTP server.

>>> from z3c.coverage.coveragediff import MailSender
>>> mailer = MailSender('smtp.example.com', 25)

Since it wouldn’t be a good idea to actually send emails from the test suite, we’ll use a stub SMTP connection class. Also, let’s hide the real one as an insurance, so that even if our stub fails, the rest of the tests won’t send any real emails:

>>> MailSender.connection_class = None

>>> class FakeSMTP(object):
...     def __init__(self, host, port):
...         print "Connecting to %s:%s" % (host, port)
...     def sendmail(self, sender, recipients, body):
...         from smtplib import quoteaddr
...         print "MAIL FROM:%s" % quoteaddr(sender)
...         if isinstance(recipients, basestring):
...             recipients = [recipients]
...         for recipient in recipients:
...             print "RCPT TO:%s" % quoteaddr(recipient)
...         print "DATA"
...         print body
...         print "."
...     def quit(self):
...         print "QUIT"
>>> mailer.connection_class = FakeSMTP

Here’s how you send an email:

>>> mailer.send_email('Some Bot <[e-mail]>',
...                   'Maintainer <[e-mail]>',
...                   'Test coverage regressions',
...                   'You broke the tests completely.  Have a nice day.')
Connecting to smtp.example.com:25
MAIL FROM:<[e-mail]>
RCPT TO:<[e-mail]>
DATA
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: Some Bot <[e-mail]>
To: Maintainer <[e-mail]>
Subject: Test coverage regressions
<BLANKLINE>
You broke the tests completely.  Have a nice day.
.
QUIT

Small utilities

There are several small utility functions like strip, urljoin, matches and filter_files. These are described (and tested) adequately by their doctests.

ReportPrinter

The ReportPrinter class is responsible for formatting the output.

>>> from z3c.coverage.coveragediff import ReportPrinter
>>> printer = ReportPrinter()
>>> printer.warn('/tmp/coverage/z3c.coverage.coveragediff.cover',
...              '3 new untested lines')
z3c.coverage.coveragediff: 3 new untested lines
>>> printer.warn('/tmp/coverage/z3c.coverage.coveragereport.cover',
...              '2 new untested lines')
z3c.coverage.coveragereport: 2 new untested lines

Links to web pages

Report printer can also include links to web pages with the coverage reports (e.g. the ones you can get from the coverage tool distributed with z3c.coverage).

>>> printer = ReportPrinter(web_url='http://example.com/coverage/')
>>> printer.warn('/tmp/coverage/z3c.coverage.coveragediff.cover',
...              '3 new untested lines')
z3c.coverage.coveragediff: 3 new untested lines
See http://example.com/coverage/z3c.coverage.coveragediff.html
<BLANKLINE>
>>> printer.warn('/tmp/coverage/z3c.coverage.coveragereport.cover',
...              '2 new untested lines')
z3c.coverage.coveragereport: 2 new untested lines
See http://example.com/coverage/z3c.coverage.coveragereport.html
<BLANKLINE>

ReportEmailer

The ReportEmailer class is an alternative to ReportPrinter. It collects warnings and sends them via email.

You pass the basic email parameters (sender, recipient and subject line) to the constructor:

>>> from z3c.coverage.coveragediff import ReportEmailer
>>> emailer = ReportEmailer('Some Bot <[e-mail]>',
...                         'Maintainer <[e-mail]>',
...                         'Test coverage regressions')

You add warnings about Python modules by passing the filename of the coverage file and the message

>>> emailer.warn('/tmp/coverage/z3c.coverage.coveragediff.cover',
...              '3 new untested lines')
>>> emailer.warn('/tmp/coverage/z3c.coverage.coveragereport.cover',
...              '2 new untested lines')

Finally you send the email. Since it wouldn’t be a good idea to actually send emails from the test suite, we’ll use a stub MailSender class:

>>> class FakeMailSender(object):
...     def send_email(self, from_addr, to_addr, subject, body):
...         print "From:", from_addr
...         print "To:", to_addr
...         print "Subject:", subject
...         print "---"
...         print body
>>> emailer.mailer = FakeMailSender()
>>> emailer.send()
From: Some Bot <[e-mail]>
To: Maintainer <[e-mail]>
Subject: Test coverage regressions
---
z3c.coverage.coveragediff: 3 new untested lines
z3c.coverage.coveragereport: 2 new untested lines

Links to web pages

Report emailer can also include links to web pages with the coverage reports (e.g. the ones you can get from the coverage tool distributed with z3c.coverage).

>>> emailer = ReportEmailer('Some Bot <[e-mail]>',
...                         'Maintainer <[e-mail]>',
...                         'Test coverage regressions',
...                         web_url='http://example.com/coverage',
...                         mailer=FakeMailSender())
>>> emailer.warn('/tmp/coverage/z3c.coverage.coveragediff.cover',
...              '3 new untested lines')
>>> emailer.warn('/tmp/coverage/z3c.coverage.coveragereport.cover',
...              '2 new untested lines')
>>> emailer.send()
From: Some Bot <[e-mail]>
To: Maintainer <[e-mail]>
Subject: Test coverage regressions
---
z3c.coverage.coveragediff: 3 new untested lines
See http://example.com/coverage/z3c.coverage.coveragediff.html
<BLANKLINE>
z3c.coverage.coveragereport: 2 new untested lines
See http://example.com/coverage/z3c.coverage.coveragereport.html
<BLANKLINE>

Empty reports

Empty reports are not sent out.

>>> emailer = ReportEmailer('Some Bot <[e-mail]>',
...                         'Maintainer <[e-mail]>',
...                         'Test coverage regressions',
...                         mailer=FakeMailSender())
>>> emailer.send()

Main function

A traditional main function parses command-line arguments and hooks up compare_dirs with the appropriate reporter.

>>> import sys
>>> from z3c.coverage.coveragediff import main

>>> def run(args):
...     try:
...         old_stderr = sys.stderr
...         sys.argv = args
...         sys.stderr = sys.stdout
...         try:
...             main()
...         finally:
...             sys.stderr = old_stderr
...     except SystemExit, e:
...         if e.code:
...             print "(returned exit code %s)" % e.code

Help message

>>> run(['coveragediff', '--help'])
Usage: coveragediff [options] olddir newdir
<BLANKLINE>
Options:
  -h, --help         show this help message and exit
  --include=REGEX    only consider files matching REGEX
  --exclude=REGEX    ignore files matching REGEX
  --email=ADDR       send the report to a given email address (only if
                     regressions were found)
  --from=ADDR        set the email sender address
  --subject=SUBJECT  set the email subject
  --web-url=BASEURL  include hyperlinks to HTML-ized coverage reports at a
                     given URL

Missing arguments

>>> run(['coveragediff'])
Usage: coveragediff [options] olddir newdir
<BLANKLINE>
coveragediff: error: wrong number of arguments
(returned exit code 2)

>>> run(['coveragediff', 'somedir'])
Usage: coveragediff [options] olddir newdir
<BLANKLINE>
coveragediff: error: wrong number of arguments
(returned exit code 2)

Excess arguments

>>> run(['coveragediff', 'dir1', 'dir2', 'dir3'])
Usage: coveragediff [options] olddir newdir
<BLANKLINE>
coveragediff: error: wrong number of arguments
(returned exit code 2)

Regular run

coveragediff follows the hallowed Unix tradition and does not print any unnecessary output, just the basics

>>> run(['coveragediff', sampleinput_dir, another_dir])
z3c.coverage.coveragediff: 36 new lines of untested code
z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)

It means that if you have no coverage regressions in your test suite, the output will be empty

>>> run(['coveragediff', another_dir, another_dir])

Include/exclude patterns

>>> run(['coveragediff', sampleinput_dir, another_dir,
...      '--include', 'fake'])
z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)

>>> run(['coveragediff', sampleinput_dir, another_dir,
...      '--exclude', 'fake'])
z3c.coverage.coveragediff: 36 new lines of untested code

Links to web pages

If you use coveragereport to produce HTML versions of the plain-text coverage files, and you have those available on the web, you can ask coveragediff to include links to the appropriate modules for convenient copy and paste (or clickety-clicking for those of us who use superior terminal emulators like GNOME Terminal).

>>> run(['coveragediff', sampleinput_dir, another_dir,
...      '--web-url', 'http://example.com/coverage'])
z3c.coverage.coveragediff: 36 new lines of untested code
See http://example.com/coverage/z3c.coverage.coveragediff.html
<BLANKLINE>
z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)
See http://example.com/coverage/z3c.coverage.fakenewmodule.html
<BLANKLINE>

Reports via email

You can ask for the output to be emailed instead of being printed to stdout.

>>> MailSender.connection_class = FakeSMTP
>>> run(['coveragediff', sampleinput_dir, another_dir,
...      '--email', 'Project List <[e-mail]>',
...      '--from', 'Coverage Daemon <[e-mail]>'])
Connecting to localhost:25
MAIL FROM:<[e-mail]>
RCPT TO:<[e-mail]>
DATA
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: Coverage Daemon <[e-mail]>
To: Project List <[e-mail]>
Subject: Unit test coverage regression
<BLANKLINE>
z3c.coverage.coveragediff: 36 new lines of untested code
z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)
.
QUIT

coveragediff.py is a script

For convenience you can download the coveragediff.py module and run it as a script

>>> sys.argv = ['coveragediff', sampleinput_dir, another_dir]
>>> script_file = os.path.join(z3c.coverage.__path__[0], 'coveragediff.py')
>>> execfile(script_file, dict(__name__='__main__'))
z3c.coverage.coveragediff: 36 new lines of untested code
z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)

CHANGES

1.1.3 (2009-07-24)

• Bug: Doctest did not normalize the whitespace in coveragediff.txt. For some reason it passes while testing independently, but when running all tests, it failed.

1.1.2 (2008-04-14)

• Bug: When a package path contained anywhere the word “test”, it was ignored from the coverage report. The intended behavior, however, was to ignore files that relate to setting up tests.

• Bug: Sort the results of os.listdir() in README.txt to avoid non-deterministic failures.

• Bug: The logic for ignoring unit and functional test modules also used to ignore modules and packages called testing.

• Change “Unit test coverage” to “Test coverage” in the title – it works perfectly fine for functional tests too.

1.1.1 (2008-01-31)

• Bug: When the package was released, the test which tests the availability of an SVN revision number failed. Made the test more reliable.

1.1.0 (2008-01-29)

• Feature: The main() coverage report function now accepts the arguments of the script as a function argument, making it easier to configure the script from buildout.

• Feature: When the report directory does not exist, the report generator creates it for you.

• Feature: Eat your own dog food by creating a buildout that can create coverage reports.

• Bug: Improved the test coverage to 100%.

1.0.1 (2007-09-26)

• Bug: Fixed meta-data.

1.0.0 (2007-09-26)

• First public release.

0.2.1

• Feature: Added the --web option to coveragediff.

• Feature: Added a test suite.

0.2.0

• Feature: Added coveragediff.py.

0.1.0

• Initial release of coveragereport.py.