pytest-phmdoctest 0.0.3

pytest plugin to test Python examples in Markdown using phmdoctest.

pytest-phmdoctest 0.0.3

Introduction

Python syntax highlighted Markdown doctest pytest plugin.

A pytest plugin based on the phmdoctest command line tool.

If you have Python syntax highlighted examples in Markdown like this:

Python code...

print("Hello World!")

plus expected output.

Hello World!

and Python interactive sessions described by doctest ...

>>> import math
>>> math.floor(9.1)
9

This pytest plugin will test them, as is, without edits. On this file try the command ...

pytest -v --phmdoctest README.md

pytest console output ...

plugins: phmdoctest-0.0.3
collected 2 items

::README.py::README.session_00001_line_24 PASSED
::README.py::test_code_14_output_18 PASSED

• No blank line is needed after the doctest output: "9".
• On GitHub view top level README.md with the Raw mode to see the Python examples in fenced code blocks. Look for the triple back tick fence at the start of the line.

Description

The plugin is based on the Python tool phmdoctest version 1.3. It generates a pytest test file from a Markdown file.

• Reads these from Markdown fenced code blocks:
  • Python interactive sessions described by doctest.
  • Python source code and expected terminal output.
• Simple use case is possible with no Markdown edits.
• More features selected by adding HTML comment directives to the Markdown. See Directives in phmdoctest.

main branch status

Website | Docs | Repos | pytest | Codecov | License

Introduction | Description | Installation | Usage | Help | phmdoctest-collect | Saving test files | Hints | Related projects

Changes | Contributions

Installation

It is advisable to install in a virtual environment.

python -m pip install pytest-phmdoctest

Usage

Consider a project with the following files:

CONTRIBUTING.md
README.md
doc/README.md
doc/directive2.md
doc/nocode.md
doc/project.md
tests/test_example.py

pytest -v

plugins: phmdoctest-0.0.3
collected 1 item

tests/test_example.py::test_example PASSED

Use --phmdoctest to collect Markdown files.

pytest --phmdoctest -v

plugins: phmdoctest-0.0.3
collected 11 items

::README.py::README.session_00001_line_24 PASSED
::README.py::test_code_10_output_17 PASSED
::doc__directive2.py::test_code_25_output_32 PASSED
::doc__directive2.py::test_code_42_output_47 PASSED
::doc__directive2.py::test_code_52_output_56 PASSED
::doc__nocode.py::test_nothing_passes PASSED
::doc__project.py::doc__project.session_00001_line_31 PASSED
::doc__project.py::doc__project.session_00002_line_46 PASSED
::doc__project.py::doc__project.session_00003_line_55 PASSED
::doc__project.py::test_code_12_output_19 PASSED
tests/test_example.py::test_example PASSED

• The sample project above can be viewed on GitHub at tests/sample.
• The doc__ indicates the Markdown file was collected from the the doc folder.
• The plugin does not need the pytest option --doctest-modules.
• You can add --phmdoctest to the addopts section of a pytest configuration ini file. Pick one.

# pytest.ini and tox.ini
[pytest]
addopts = --phmdoctest

# setup.cfg
[tool:pytest]
addopts = --phmdoctest

# pyproject.toml
[tool.pytest.ini_options]
addopts = "--phmdoctest"

Markdown ".md" files are discovered by pytest. pytest finds them in the same way it finds Python test files. For each Markdown file discovered, the plugin generates a pytest test file. The file is saved and collected from a temporary directory managed by the plugin. Some well known Markdown files get ignored automatically.

• README.md except at the repository root.
• All uppercase names like CONTRIBUTING.md.

If the Markdown file does not have any fenced code block examples a test file with the test case named test_nothing_passes() is generated and collected.

To avoid collecting .md files use pytest --ignore and --ignore-glob on the command line or in the addopts part of the pytest ini file. These commands work on .md files and use Unix shell-style wildcards.

# With a terminal in the tests/sample directory
# The first line collects 6 items.
# The second line collects 3 items.

pytest -v --phmdoctest --ignore README.md --ignore doc/directive2.md
pytest -v --phmdoctest --ignore-glob */*.md

Help

pytest --help contains a phmdoctest: group in the middle and an ini-option near the bottom. The help contains:

• --phmdoctest
• --phmdoctest-save
• phmdoctest-collect

phmdoctest-collect

An optional phmdoctest-collect = section can be placed in the pytest ini file. It is a list of lines of the format

glob [phmdoctest command line options]

• The Markdown file must match one of the globs.
• The glob is processed by Path.glob() from the Python standard library pathlib. Path.glob() offers a "**" recursive pattern that means “this directory and all subdirectories, recursively”.
• The globs are checked from top to bottom. The first glob to match the Markdown file determines the phmdoctest command line options.
• If there is no match the file will not be collected.
• No files are auto ignored.
• A line can have just the glob and no options. The glob is required.
• The options should look like and have the same spacing as the command line options passed to the tool phmdoctest.
  • Use double quotes as needed in TEXT.
  • The plugin does not support \" escaped double quote.
  • Look for list of options in the next section.
• If a line that does not parse is needed, the plugin collects a special test file that contains a failing test case with an embedded error message.

Example

# pytest.ini
[pytest]
addopts = --phmdoctest
phmdoctest-collect =
    doc/project.md
    **/*code.md --fail-nocode

Then run this pytest command on the project files from the Usage section ...

pytest -v --ignore tests/test_example.py

output

plugins: phmdoctest-0.0.3
collected 5 items

::doc__nocode.py::test_nothing_fails FAILED
::doc__project.py::doc__project.session_00001_line_31 PASSED
::doc__project.py::doc__project.session_00002_line_46 PASSED
::doc__project.py::doc__project.session_00003_line_55 PASSED
::doc__project.py::test_code_12_output_19 PASSED

• The glob on the first line matches the file doc/project.md generating the last 4 items in the output.
• The glob on the second line matches doc/nocode.md. The file does not have any examples. The phmdoctest option --fail-nocode tells phmdoctest to generate the test test_nothing_fails. That test results in the FAILED status of the first item collected.
• The ini file globs above only apply to .md files. We ignore the Python pytest test file tests/test_example.py by adding the --ignore option.

phmdoctest-collect options

-s, --skip TEXT

Do not test blocks with substring TEXT. Allowed multiple times.

--fail-nocode

Markdown file with no code blocks generates a failing test.

-u, --setup TEXT

Run block with substring TEXT at test module setup time.

-d, --teardown TEXT

Run block with substring TEXT at test module teardown time.

--setup-doctest

Make globals created by the --setup Python code block or setup directive visible to Python interactive session >>> blocks. Caution: The globals are set at pytest Session scope. The globals are visible to all doctests in the test suite. This includes doctests collected by the plugin and doctests collected from other files due to --doctest-modules.

Saving test files

The generated pytest files can be saved for later inspection using the --phmdoctest-save pytest command line option. Specify a target directory for the files.

• The saved files have the prefix test_.
• If the directory exists it will be used, but not cleaned.
• If the directory does not exist, it is created.
• The plugin does not create a temporary directory.
• The plugin does not collect any files.
• pytest has been observed to collect test_*.py files saved to a pre-existing directory in the same invocation.

Hints

• When invoking pytest, cwd must be in the subpath of the files to be collected to avoid this error from pathlib.py in relative_to(): ValueError: <file to be collected> is not in the subpath of <working directory>
• Note the plugin does not accept single quoted phmdoctest args in the phmdoctest-collect section. A failing test will be collected.
• Use underscore in conftest.py for pytest_plugins: pytest_plugins = ["pytest_phmdoctest"]
• An ImportPathMismatchError indicates two test files have the same name.
• pytest -vv output shows the path to the plugin temporary directory.
• The --report option of the phmdoctest command lists all fenced code blocks in the Markdown file.

Related projects

• phmdoctest
• rundoc
• byexample
• sphinx.ext.doctest
• sybil
• doxec
• egtest
• pytest-codeblocks

MIT License

Copyright (c) 2021 Mark Taylor

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.