phmdoctest 0.1.0

Python syntax highlighted Markdown doctest.

phmdoctest 0.1.0

Introduction

Python syntax highlighted Markdown doctest

Command line program to test Python syntax highlighted code examples in Markdown.

• Synthesizes a pytest test file from examples in Markdown.
• Reads these from Markdown fenced code blocks:
  • Python interactive sessions described by doctest.
  • Python source code and expected terminal output.
• No extra tags or html comments needed in the Markdown. No Markdown edits at all.
• The test cases are run later by calling pytest.
• Get code coverage by running pytest with coverage.
• An included Python library: Development tools API.
  • runs phmdoctest and can run pytest too. (simulator.py)
  • functions to read fenced code blocks from Markdown. (tool.py)

phmdoctest does not do setup and teardown. Each test case runs independently.

master branch status

Documentation | Homepage | Build | Codecov | License

Installation

It is advisable to install in a virtual environment.

python -m pip install phmdoctest

Sample usage

Given the Markdown file example1.md shown in raw form here...

# This is Markdown file example1.md

## Interactive Python session (doctest)

```pycon 
>>> print('Hello World!')
Hello World!
```

## Source Code and terminal output

Code:
```python3
from enum import Enum

class Floats(Enum):
    APPLES = 1
    CIDER = 2
    CHERRIES = 3
    ADUCK = 4
for floater in Floats:
    print(floater)
```

sample output:
```
Floats.APPLES
Floats.CIDER
Floats.CHERRIES
Floats.ADUCK
```

the command...

phmdoctest doc/example1.md --outfile test_example1.py

creates the python source code file test_example1.py shown here...

"""pytest file built from doc/example1.md"""
from itertools import zip_longest


def line_by_line_compare_exact(a, b):
    """Line by line helper compare function with assertion for pytest."""
    a_lines = a.splitlines()
    b_lines = b.splitlines()
    for a_line, b_line in zip_longest(a_lines, b_lines):
        assert a_line == b_line


def session_00001_line_6():
    r"""
    >>> print('Hello World!')
    Hello World!
    """


def test_code_14_output_27(capsys):
    from enum import Enum

    class Floats(Enum):
        APPLES = 1
        CIDER = 2
        CHERRIES = 3
        ADUCK = 4
    for floater in Floats:
        print(floater)

    expected_str = """\
Floats.APPLES
Floats.CIDER
Floats.CHERRIES
Floats.ADUCK
"""
    line_by_line_compare_exact(a=expected_str, b=capsys.readouterr().out)

Then run a pytest command something like this in your terminal to test the Markdown session, code, and expected output blocks.

pytest --strict --doctest-modules

Or these two commands:

pytest --strict
python -m doctest test_example1.py

The line_6 in the function name session_00001_line_6 is the line number in example1.md of the first line of the interactive session. 00001 is a sequence number to order the doctests.

The 14 in the function name test_code_14_output_27 is the line number of the first line of python code. 27 shows the line number of the expected terminal output.

phmdoctest tries to generate one test case function for each:

• Markdown fenced code block interactive session
• Python-code/expected-output Markdown fenced code block pair

The --report option below shows the blocks discovered and how phmdoctest will test them.

--report option

To see the GFM fenced code blocks in the MARKDOWN_FILE use the --report option like this:

phmdoctest doc/example2.md --report

which lists the fenced code blocks it found in the file example2.md. The test role column shows how phmdoctest will test each fenced code block.

         doc/example2.md fenced blocks
-----------------------------------------------
block    line  test     skip pattern/reason
type   number  role     quoted and one per line
-----------------------------------------------
py3         9  code
           14  output
py3        20  code
           26  output
           31  --
py3        37  code
py3        44  code
           51  output
yaml       59  --
text       67  --
py         72  session
py3        80  code
           86  output
pycon      94  session
-----------------------------------------------
7 test cases
1 code blocks missing an output block

How phmdoctest identifies code, session, and output blocks

phmdoctest uses the PYPI commonmark project to extract fenced code blocks from Markdown. Specification CommonMark Spec and website CommonMark.

Only GFM fenced code blocks are considered.

A block is a session block if the info_string starts with 'py' and the first line of the block starts with the session prompt: '>>> '.

To be treated as Python code the opening fence should start with one of these:

```python
```python3
```py3

and the block contents can't start with '>>> '.

project.md has more examples of code and session blocks.

It is ok if the info string is laden with additional text, phmdoctest will ignore it. The entire info string will be shown in the block type column of the report.

Output blocks are fenced code blocks that immediately follow a Python block and start with an opening fence like this which has an empty info string.

```

If a Python code block has no output if it is followed by any of:

• Python code block
• Python session block
• a fenced code block with a non-empty info string

phmdoctest will still generate test code for it, but there will be no assertion statement.

Skipping Python blocks with the --skip option

If you don't want to generate test cases for Python blocks use the --skip TEXT option. More than one --skip TEXT is allowed.

The code in each Python block is searched for the substring TEXT. Zero, one or more blocks will contain the substring. These blocks will not generate test cases in the output file.

• The Python code in the fenced code block is searched.
• The info string is not searched.
• Output blocks are not searched.
• Only Python code or session blocks are searched.
• Case is significant.

The report shows which Python blocks are skipped in the test role column and the Python blocks that matched each --skip TEXT in the skips section.

This option makes it very easy to inadvertently exclude Python blocks from the test cases. In the event no test cases are generated, the option --fail-nocode described below is useful.

Three special --skip TEXT strings work a little differently. They select one of the first, second, or last of the Python blocks. Only Python blocks are counted.

• --skip FIRST skips the first Python block
• --skip SECOND skips the second Python block
• --skip LAST skips the final Python block

--skip Example

This command

phmdoctest doc/example2.md --skip "Python 3.7" --skip LAST --report --outfile test_example2.py

Produces the report

           doc/example2.md fenced blocks
----------------------------------------------------
block    line  test          skip pattern/reason
type   number  role          quoted and one per line
----------------------------------------------------
py3         9  code
           14  output
py3        20  skip-code     "Python 3.7"
           26  skip-output
           31  --
py3        37  code
py3        44  code
           51  output
yaml       59  --
text       67  --
py         72  session
py3        80  code
           86  output
pycon      94  skip-session  "LAST"
----------------------------------------------------
5 test cases
1 skipped code blocks
1 skipped interactive session blocks
1 code blocks missing an output block

  skip pattern matches (blank means no match)
------------------------------------------------
skip pattern  matching code block line number(s)
------------------------------------------------
Python 3.7    20
LAST          94
------------------------------------------------

and creates the output file test_example2.py

-s short option form of --skip

This is the same command as above using the short -s form of the --skip option in two places. It produces the same report and outfile.

phmdoctest doc/example2.md -s "Python 3.7" -sLAST --report --outfile test_example2.py

--fail-nocode

This option produces a pytest file that will always fail when no Python code or session blocks are found.

If phmdoctest doesn't find any Python code or session blocks in the Markdown file a pytest file is still generated. This also happens when --skip eliminates all the Python code blocks. The generated pytest file will have the function def test_nothing_passes().

If the option --fail-nocode is passed to phmdoctest the function is def test_nothing_fails() which raises an assertion.

Send outfile to standard output

To redirect the above outfile to the standard output stream use one of these two commands.

Be sure to leave out --report when sending --outfile to standard output.

phmdoctest doc/example2.md -s "Python 3.7" -sLAST --outfile -

or

phmdoctest doc/example2.md -s "Python 3.7" -sLAST --outfile=-

Usage

phmdoctest --help

Usage: phmdoctest [OPTIONS] MARKDOWN_FILE

Options:
  --outfile TEXT   Write generated test case file to path TEXT. "-" writes to
                   stdout.

  -s, --skip TEXT  Any Python code or interactive session block that contains
                   the substring TEXT is not tested. More than one --skip TEXT
                   is ok. Double quote if TEXT contains spaces. For example
                   --skip="python 3.7" will skip every Python block that
                   contains the substring "python 3.7". If TEXT is one of the
                   3 capitalized strings FIRST SECOND LAST the first, second,
                   or last Python block in the Markdown file is skipped. The
                   fenced code block info string is not searched.

  --report         Show how the Markdown fenced code blocks are used.
  --fail-nocode    This option sets behavior when the Markdown file has no
                   Python fenced code blocks or interactive session blocks or
                   if all such blocks are skipped. When this option is present
                   the generated pytest file has a test function called
                   test_nothing_fails() that will raise an assertion. If this
                   option is not present the generated pytest file has
                   test_nothing_passes() which will never fail.

  --version        Show the version and exit.
  --help           Show this message and exit.

Running on Travis CI

The partial script shown below is for Python 3.5 on Travis CI. The script steps are:

• Install phmdoctest (the ".") and install pytest.
• Create a new directory to take the generated test file.
• Run phmdoctest to generate the test file and print the report.
• Run pytest suite.

Writing the generated test files to a new directory assures an existing test file is not overwritten by mistake.

dist: xenial
language: python
sudo: false

matrix:
  include:
    - python: 3.5
      install:
        - pip install "." pytest
      script:
        - mkdir tests/tmp
        - phmdoctest project.md --report --outfile tests/tmp/test_project.py
        - pytest --strict --doctest-modules -vv tests

Running phmdoctest from the command line as a Python module

Here is an example:

python -m phmdoctest doc/example2.md --report

Testing phmdoctest from within a Python script

phmdoctest.simulator offers the function run_and_pytest() which simulates running phmdoctest from the command line.

• useful during development
• creates the --outfile in a temporary directory
• optionally runs pytest on the outfile

Please see the Development tools API section or the docstring of the function run_and_pytest() in the file simulator.py. pytest_options are passed as a list of strings as shown below.

import phmdoctest.simulator
command = 'phmdoctest doc/example1.md --report --outfile test_me.py'
simulator_status = phmdoctest.simulator.run_and_pytest(
    well_formed_command=command,
    pytest_options=['--strict', '--doctest-modules', '-v']
)
assert simulator_status.runner_status.exit_code == 0
assert simulator_status.pytest_exit_code == 0

Execution Context

• Interactive sessions run in the doctest execution context.
• Code/expected output run within a function body of a pytest test case.
• Pytest and doctest determine the order of test case execution.

Hints

• phmdoctest can read the Markdown file from the standard input stream. Use - for MARKDOWN_FILE.
• Write the test file to a temporary directory so that it is always up to date.
• Its easy to use --output by mistake instead of --outfile.
• If Python code block has no output, put assert statements in the code.
• Use pytest option --doctest-modules to test the sessions.
• phmdoctest ignores Markdown indented code blocks (Spec section 4.4).
• simulator_status.runner_status.exit_code == 2 is the click command line usage error.
• Since phmdoctest generates code, the input file should be from a trusted source.

Related projects

• rundoc
• byexample
• sybil
• doexec
• egtest

Recent Changes