pytest-markdown-console 0.0.1.dev2

A pytest extension to test console code blocks in markdown files.

A pytest extension to test console code blocks in Markdown files

A minimal pytest extension to test console blocks in your Markdown documentation. Works on Linux, macOS, and Windows.

Installation

In your project, run:

pip install pytest-markdown-console

If you are using uv, install it as a development dependency:

uv add --dev pytest-markdown-console

The plugin self-registers and will run in each subsequent pytest invocation.

How does it work?

Each console code block in Markdown files collected by the plugin generates a test case.

The test runs each command in the block and compares its actual output to the expected output written in the block.

Take this Markdown file:

# README.md

Here is an example of running my app:

```console
$ uv run myapp.py
Hello, world!
```

Running pytest with this extension creates a test case that passes if the app actually prints "Hello, world!".

Match partial output

Use ... anywhere in the expected output to match any text in its place, including multiple lines:

```console
$ python -c "print(object())"
<object object at 0x...>
```

This is useful when part of the output is non-deterministic, such as memory addresses or timestamps, or when the output is very long.

Signal expected failures

Sometimes you want to illustrate expected failures. To convey this to both the reader and the plugin, place a # Error: comment at the end of the failing command, or on the line preceding it:

```console
$ uv run myapp.py --bad_option  # Error: this will fail
...
$ # Error: this will also most definitely fail
$ uv run myapp.py --still_bad
...
```

To pass the test case, the command line must fail and the error message produced by the application, if any, must match.

Filter by platform

To restrict a test to specific platforms, use the platform: directive in an HTML comment immediately before the fence:

<!-- pytest-markdown-console: platform:linux,macos -->
```console
$ echo "This will generate a test case on Linux and macOS"
```

To exclude a platform, prefix its name with !:

<!-- pytest-markdown-console: platform:!windows -->
```console
$ echo "This will not generate a test case on Windows"
```

Change the working directory

By default, all commands run in the same directory as the Markdown file. To use a different directory, use the cwd: directive:

<!-- pytest-markdown-console: cwd:../ -->
```console
$ uv run myapp.py
...
```

Relative paths are resolved relative to the Markdown file's location.

Exclude a block from testing

To exclude a block from being collected as a test at all, use the notest directive:

<!-- pytest-markdown-console: notest -->
```console
$ echo "This block will not be tested"
```

Controlling test case runs globally

By default, test cases generated by this plugin are run whenever pytest is invoked. To exclude them entirely:

pytest -p no:markdown-console

To run only the plugin's tests:

pytest -m markdown_console

Customising the directive tag

By default, directive comments use the pytest-markdown-console tag:

<!-- pytest-markdown-console: notest -->
```console
$ echo "This block will not be tested"
```

You can change this tag via pyproject.toml, for example to keep your Markdown source shorter:

[tool.pytest.ini_options]
markdown_console_directive = "console-test"

With the above setting you would write <!-- console-test: notest --> instead.

Why this extension?

This plugin makes your documentation testable — specifically console blocks — within the same pytest suite you use for the rest of your Python code.

Other tools can test console blocks in Markdown files, but we couldn't find one that is simple, supports Windows, integrates with pytest, and requires no boilerplate.

Does it make sense to test console blocks?

Testing console blocks is admittedly niche. They often contain installation instructions or shell-specific commands that don't translate across platforms.

However, if you are building a CLI app, you likely already showcase commands and their output in your docs. Testing those snippets ensures they stay up-to-date.

Launching a Python app on Windows, Linux, or macOS is the same one-liner when using uv. That is the sweet spot motivating this small plugin.