manuel 1.0.0a6

Manuel lets you combine traditional doctests with new test syntax that you build yourself or is inlcuded in Manuel.

======
Manuel
======

.. contents::


Overview
========

In short, Manuel parses documents (tests), evaluates their contents, then
formats the result of the evaluation.

For a quick introduction to Manuel, take a look at the `FIT Table Example`_.

The core functionality is accessed through an instance of a Manuel object. It
is used to build up the handling of a document type. Each phase has a
corresponding slot to which various implementations are attached.

>>> import manuel


Parsing
-------

Manuel operates on Documents. Each Document is created from a string
containing one or more lines.

>>> source = """\
... This is our document, it has several lines.
... one: 1, 2, 3
... two: 4, 5, 7
... three: 3, 5, 1
... """
>>> document = manuel.Document(source)

For example purposes we will create a type of test that consists of a sequence
of numbers so lets create a NumbersTest object to represent the parsed list.

>>> class NumbersTest(object):
... def __init__(self, description, numbers):
... self.description = description
... self.numbers = numbers

The Document is divided into one or more regions. Each region is a distinct
"chunk" of the document and will be acted uppon in later (post-parsing) phases.
Initially the Document is made up of a single element, the source string.

>>> [region.source for region in document]
['This is our document, it has several lines.\none: 1, 2, 3\ntwo: 4, 5,
7\nthree: 3, 5, 1\n']

The Document offers a "find_regions" method to assist in locating the portions
of the document a particular parser is interested in. Given a regular
expression (either as a string, or compiled), it will return "region" objects
that contain the matched source text, the line number (1 based) the region
begins at, as well as the associated re.Match object.

>>> import re
>>> numbers_test_finder = re.compile(
... r'^(?P<description>.*?): (?P<numbers>(\d+,?[ ]?)+)$', re.MULTILINE)
>>> regions = document.find_regions(numbers_test_finder)
>>> regions
[<manuel.Region object at 0x...>,
<manuel.Region object at 0x...>,
<manuel.Region object at 0x...>]
>>> regions[0].lineno
2
>>> regions[0].source
'one: 1, 2, 3\n'
>>> regions[0].start_match.group('description')
'one'
>>> regions[0].start_match.group('numbers')
'1, 2, 3'

If given two regular expressions find_regions will use the first to identify
the begining of a region and the second to identify the end.

>>> region = document.find_regions(
... re.compile('^one:.*$', re.MULTILINE),
... re.compile('^three:.*$', re.MULTILINE),
... )[0]
>>> region.lineno
2
>>> print region.source
one: 1, 2, 3
two: 4, 5, 7
three: 3, 5, 1

Also, instead of just a "start_match" attribute, the region will have
start_match and end_match attributes.

>>> region.start_match
<_sre.SRE_Match object at 0x...>
>>> region.end_match
<_sre.SRE_Match object at 0x...>


Regions must always consist of whole lines.

>>> document.find_regions('1, 2, 3')
Traceback (most recent call last):
...
ValueError: Regions must start at the begining of a line.

>>> document.find_regions('three')
Traceback (most recent call last):
...
ValueError: Regions must end at the ending of a line.

>>> document.find_regions(
... re.compile('ne:.*$', re.MULTILINE),
... re.compile('^one:.*$', re.MULTILINE),
... )
Traceback (most recent call last):
...
ValueError: Regions must start at the begining of a line.

>>> document.find_regions(
... re.compile('^one:.*$', re.MULTILINE),
... re.compile('^three:', re.MULTILINE),
... )
Traceback (most recent call last):
...
ValueError: Regions must end at the ending of a line.

Now we can register a parser that will identify the regions we're interested in
and create NumbersTest objects from the source text.

>>> def parse(document):
... for region in document.find_regions(numbers_test_finder):
... description = region.start_match.group('description')
... numbers = map(
... int, region.start_match.group('numbers').split(','))
... test = NumbersTest(description, numbers)
... document.replace_region(region, test)

>>> parse(document)
>>> [region.source for region in document]
['This is our document, it has several lines.\n',
'one: 1, 2, 3\n',
'two: 4, 5, 7\n',
'three: 3, 5, 1\n']
>>> [region.parsed for region in document]
[None,
<NumbersTest object at 0x...>,
<NumbersTest object at 0x...>,
<NumbersTest object at 0x...>]


Evaluation
----------

After a document has been parsed the resulting tests are evaluated. Unlike
parsing and formatting, evaluation is done one region at a time, in the order
that the regions appear in the document. Manuel provides another method to
evaluate tests. Lets define a function to evaluate NumberTests. The function
determines whether or not the numbers are in sorted order and records the
result along with the description of the list of numbers.

>>> class NumbersResult(object):
... def __init__(self, test, passed):
... self.test = test
... self.passed = passed

>>> def evaluate(region, document, globs):
... if not isinstance(region.parsed, NumbersTest):
... return
... test = region.parsed
... passed = sorted(test.numbers) == test.numbers
... region.evaluated = NumbersResult(test, passed)

>>> for region in document:
... evaluate(region, document, {})
>>> [region.evaluated for region in document]
[None,
<NumbersResult object at 0x...>,
<NumbersResult object at 0x...>,
<NumbersResult object at 0x...>]


Formatting
----------

Once the evaluation phase is completed the results are formatted. You guessed
it: manuel provides a method for formatting results. We'll build one to format
a message about whether or not our lists of numbers are sorted properly. A
formatting function returns None when it has no output, or a string otherwise.

>>> def format(document):
... for region in document:
... if not isinstance(region.evaluated, NumbersResult):
... continue
... result = region.evaluated
... if not result.passed:
... region.formatted = (
... "the numbers aren't in sorted order: "
... + ', '.join(map(str, result.test.numbers)))

Since our test case passed we don't get anything out of the report function.

>>> format(document)
>>> [region.formatted for region in document]
[None, None, None, "the numbers aren't in sorted order: 3, 5, 1"]


Manuel Objects
--------------

We'll want to use these parse, evaluate, and format functions later, so we
bundle them together into a Manuel object.

>>> sorted_numbers_manuel = manuel.Manuel(
... parsers=[parse], evaluaters=[evaluate], formatters=[format])


Doctests
========

We can use Manuel to run doctests. Let's create a simple doctest to
demonstrate with.

>>> source = """This is my
... doctest.
...
... >>> 1 + 1
... 2
... """
>>> document = manuel.Document(source)

The manuel.doctest module has handlers for the various phases. First we'll
look at parsing.

>>> import manuel.doctest
>>> m = manuel.doctest.Manuel()
>>> document.parse_with(m)
>>> for region in document:
... print (region.lineno, region.parsed or region.source)
(1, 'This is my\ndoctest.\n\n')
(4, <zope.testing.doctest.Example instance at 0x...>)
(6, '\n')

Now we can evaluate the examples.

>>> document.evaluate_with(m, globs={})
>>> for region in document:
... print (region.lineno, region.evaluated or region.source)
(1, 'This is my\ndoctest.\n\n')
(4, <manuel.doctest.DocTestResult instance at 0x...>)
(6, '\n')

And format the results.

>>> document.format_with(m)
>>> document.formatted()
''

Oh, we didn't have any failing tests, so we got no output. Let's try again
with a failing test. This time we'll use the process function to simplify
things.

>>> document = manuel.Document("""This is my
... doctest.
...
... >>> 1 + 1
... 42
... """)

>>> document.process_with(m, globs={})
>>> print document.formatted()
File "<memory>", line 4, in <memory>
Failed example:
1 + 1
Expected:
42
Got:
2


Globals
-------

Even though each region is parsed into its own object, state is still shared
between them. Each region of the document is executed in order so state
changes made by earlier evaluaters are available to the current evaluator.


>>> document = manuel.Document("""
... >>> x = 1
...
... A little prose to separate the examples.
...
... >>> x
... 1
... """)
>>> document.process_with(m, globs={})
>>> print document.formatted()

Imported modules are added to the global namespace as well.

>>> document = manuel.Document("""
... >>> import string
...
... A little prose to separate the examples.
...
... >>> string.digits
... '0123456789'
...
... """)
>>> document.process_with(m, globs={})
>>> print document.formatted()


Combining Test Types
====================

Now that we have both doctests and the silly "sorted numbers" tests, lets
create a single document that has both.

>>> document = manuel.Document("""
... We can test Python...
...
... >>> 1 + 1
... 42
...
... ...and lists of numbers.
...
... a very nice list: 3, 6, 2
... """)

Obviously both of those tests will fail, but first we have to configure Manuel
to understand both test types. We'll start with a doctest configuration and add
the number list testing on top.

>>> m = manuel.doctest.Manuel()

Since we already have a Manuel instance configured for our "sorted numbers"
tests, we can extend the built-in doctest configuration with it.

>>> m.extend(sorted_numbers_manuel)

Now we can process our source that combines both types of tests and see what
we get.

>>> document.process_with(m, globs={})

The document was parsed and has a mixture of prose and parsed doctests and
number tests.

>>> for region in document:
... print (region.lineno, region.parsed or region.source)
(1, '\nWe can test Python...\n\n')
(4, <doctest.Example instance at 0x...>)
(6, '\n...and lists of numbers.\n\n')
(9, <NumbersTest object at 0x...>)

We can look at the formatted output to see that each of the two tests failed.

>>> for region in document:
... if region.formatted:
... print '-'*70
... print region.formatted,
----------------------------------------------------------------------
File "<memory>", line 4, in <memory>
Failed example:
1 + 1
Expected:
42
Got:
2
----------------------------------------------------------------------
the numbers aren't in sorted order: 3, 6, 2


Priorities
==========

Some functionality requires that code be called early or late in a phase. The
"timing" decorator allows either EARLY or LATE to be specified.

Early functions are run first (in arbitrary order), then functions with no
specified timing, then the late functions are called (again in arbitrary
order). This function also demonstrates the "copy" method of Region objects
and the "insert_region_before" and "insert_region_after" methods of Documents.

>>> @manuel.timing(manuel.LATE)
... def cloning_parser(document):
... to_be_cloned = None
... # find the region to clone
... document_iter = iter(document)
... for region in document_iter:
... if region.parsed:
... continue
... if region.source.strip().endswith('my clone:'):
... to_be_cloned = document_iter.next().copy()
... break
... # if we found the region to cloned, do so
... if to_be_cloned:
... # make a copy since we'll be mutating the document
... for region in list(document):
... if region.parsed:
... continue
... if 'clone before *here*' in region.source:
... clone = to_be_cloned.copy()
... clone.provenance = 'cloned to go before'
... document.insert_region_before(region, clone)
... if 'clone after *here*' in region.source:
... clone = to_be_cloned.copy()
... clone.provenance = 'cloned to go after'
... document.insert_region_after(region, clone)

>>> m.add_parser(cloning_parser)

>>> source = """\
... This is my clone:
...
... clone: 1, 2, 3
...
... I want some copies of my clone.
...
... For example, I'd like a clone before *here*.
...
... I'd also like a clone after *here*.
... """
>>> document = manuel.Document(source)
>>> document.process_with(m, globs={})
>>> [(r.source, r.provenance) for r in document]
[('This is my clone:\n\n', None),
('clone: 1, 2, 3\n', None),
('clone: 1, 2, 3\n', 'cloned to go before'),
("\nI want some copies of my clone.\n\nFor example, I'd like a clone before
*here*.\n\nI'd also like a clone after *here*.\n", None),
('clone: 1, 2, 3\n', 'cloned to go after')]


FIT Table Example
=================

Occasionally when structuring a doctest, you want to succinctly express several
sets of inputs and outputs of a particular function.

That's something `FIT <http://fit.c2.com/wiki.cgi?SimpleExample>`_ tables do a
good job of.

Lets write a simple table evaluator using Manuel.

We'll use `reST <http://docutils.sourceforge.net/rst.html>`_ tables. The table
source will look like this:

===== ===== ======
\ A or B
--------------------
A B Result
===== ===== ======
False False False
True False True
False True True
True True True
===== ===== ======

When rendered, it will look like this:

===== ===== ======
\ A or B
--------------------
A B Result
===== ===== ======
False False False
True False True
False True True
True True True
===== ===== ======

Lets imagine that the source of our test was stored in a string:

>>> source = """\
... The "or" operator
... =================
...
... Here is an example of the "or" operator in action:
...
... ===== ===== ======
... \ A or B
... --------------------
... A B Result
... ===== ===== ======
... False False False
... True False True
... False True True
... True True True
... ===== ===== ======
... """


Parsing
-------

First we need a function to find the tables:

.. code-block:: python

import re

table_start = re.compile(r'(?<=\n\n)=[= ]+\n(?=[ \t]*?\S)', re.DOTALL)
table_end = re.compile(r'\n=[= ]+\n(?=\Z|\n)', re.DOTALL)

class Table(object):
def __init__(self, expression, variables, examples):
self.expression = expression
self.variables = variables
self.examples = examples

def parse_tables(document):
for region in document.find_regions(table_start, table_end):
lines = enumerate(iter(region.source.splitlines()))
lines.next() # skip the first line

# grab the expression to be evaluated
expression = lines.next()[1]
if expression.startswith('\\'):
expression = expression[1:]

lines.next() # skip the divider line
variables = [v.strip() for v in lines.next()[1].split()][:-1]

lines.next() # skip the divider line

examples = []
for lineno_offset, line in lines:
if line.startswith('='):
break # we ran into the final divider, so stop

values = [eval(v.strip(), {}) for v in line.split()]
inputs = values[:-1]
output = values[-1]

examples.append((inputs, output, lineno_offset))

table = Table(expression, variables, examples)
document.replace_region(region, table)

Instances of the class "Table" will represent the tables.

Now we can create a Manuel Document from the source and use the "parse_tables"
function on it.

>>> import manuel
>>> document = manuel.Document(source, location='fake.txt')
>>> parse_tables(document)

If we examine the Docuement object we can see that the table was recognized.

>>> region = list(document)[1]
>>> print region.source,
===== ===== ======
\ A or B
--------------------
A B Result
===== ===== ======
False False False
True False True
False True True
True True True
===== ===== ======

>>> region.parsed
<Table object at ...>


Evaluating
==========

Now that we can find and extract the tables from the source, we need to be able
to evaluate them.

.. code-block:: python

class TableErrors(list):
pass


class TableError(object):
def __init__(self, location, lineno, expected, got):
self.location = location
self.lineno = lineno
self.expected = expected
self.got = got

def __str__(self):
return '<%s %s:%s>' % (
self.__class__.__name__, self.location, self.lineno)


def evaluate_table(region, document, globs):
if not isinstance(region.parsed, Table):
return

table = region.parsed
errors = TableErrors()
for inputs, output, lineno_offset in table.examples:
result = eval(table.expression, dict(zip(table.variables, inputs)))
if result != output:
lineno = region.lineno + lineno_offset
errors.append(
TableError(document.location, lineno, output, result))

region.evaluated = errors

Now the table can be evaluated:

>>> evaluate_table(region, document, {})

Yay! There were no errors:

>>> region.evaluated
[]

What would happen if there were?

>>> source_with_errors = """\
... The "or" operator
... =================
...
... Here is an (erroneous) example of the "or" operator in action:
...
... ===== ===== ======
... \ A or B
... --------------------
... A B Result
... ===== ===== ======
... False False True
... True False True
... False True False
... True True True
... ===== ===== ======
... """

>>> document = manuel.Document(source_with_errors, location='fake.txt')
>>> parse_tables(document)
>>> region = list(document)[1]

This time the evaluator records errors:

>>> evaluate_table(region, document, {})
>>> region.evaluated
[<TableError object at ...>]


Formatting Errors
=================

Now that we can parse the tables and evaluate them, we need to be able to
display the results in a readable fashion.

.. code-block:: python

def format_table_errors(document):
for region in document:
if not isinstance(region.evaluated, TableErrors):
continue

# if there were no errors, there is nothing to report
if not region.evaluated:
continue

messages = []
for error in region.evaluated:
messages.append('%s, line %d: expected %r, got %r instead.' % (
error.location, error.lineno, error.expected, error.got))

sep = '\n '
header = 'when evaluating table at %s, line %d' % (
document.location, region.lineno)
region.formatted = header + sep + sep.join(messages)


Now we can see how the formatted results look.

>>> format_table_errors(document)
>>> print region.formatted,
when evaluating table at fake.txt, line 6
fake.txt, line 11: expected True, got False instead.
fake.txt, line 13: expected False, got True instead.


All Together Now
================

All the pieces (parsing, evaluating, and formatting) are available now, so we
just have to put them together into a single "Manuel" object.

>>> m = manuel.Manuel(parsers=[parse_tables], evaluaters=[evaluate_table],
... formatters=[format_table_errors])

Now we can create a fresh document and tell it to do all the above steps with
our Manuel instance.

>>> document = manuel.Document(source_with_errors, location='fake.txt')
>>> document.process_with(m, globs={})
>>> print document.formatted(),
when evaluating table at fake.txt, line 6
fake.txt, line 11: expected True, got False instead.
fake.txt, line 13: expected False, got True instead.

Of course, if there were no errors, nothing would be reported:

>>> document = manuel.Document(source, location='fake.txt')
>>> document.process_with(m, globs={})
>>> print document.formatted()

If we wanted to use our new table tests in a file named "table-example.txt" and
include them in a unittest TestSuite, it would look something like this:

.. code-block:: python

import unittest
import manuel.testing

suite = unittest.TestSuite()
suite = manuel.testing.TestSuite(m, 'table-example.txt')

If the tests are run (e.g., by a test runner), everything works.

>>> suite.run(unittest.TestResult())
<unittest.TestResult run=1 errors=0 failures=0>


CHANGES
=======

1.0.0a4 (2009-05-01)
--------------------

- make the global state ("globs") shared between all evaluators, not just
doctest


1.0.0a3 (2009-05-01)
--------------------

- make zope.testing's testrunner recognized the enhanced, doctest-style
errors generated by Manuel
- rework the evaluaters to work region-by-region instead of on the
entire document
- switch to using regular Python classes for Manuel objects instead of
previous prototype-y style


1.0.0a2 (2008-10-17)
--------------------

- first public release