pytest-nb-as-test 0.1.7

Use notebooks as pytests. Keep your notebooks working.

pytest-nb-as-test Plugin

In scientific codebases, notebooks are a convenient way to provide executable examples, figures, and LaTeX. However, example notebooks often become silently broken as the code evolves because developers rarely re-run them. New users then discover the breakage when they try the examples, which is disheartening and frustrating. This plugin executes notebook code cells as pytest tests, so example notebooks run in CI and stay up to date.

When to use

• You want .ipynb notebooks collected by pytest and run in CI.
• You want in process execution, so fixtures and monkeypatching apply.
• You need per cell control (skip, force run, expect exception, timeouts) via directives.

For comparison with other plugins/ projects see Prior art and related tools.

Install

install using pip

pip install pytest-nb-as-test

or add as dependency in pyproject.toml:

[project]
dependencies = [
  "pytest-nb-as-test",
]

Run

Pytest discovers all notebooks alongside normal tests:

pytest

Filter which notebooks are collected:

pytest --notebook-glob 'test_*.ipynb'

Disable notebook collection and execution:

pytest -p no:pytest_nb_as_test

Cell directives

Directives live in comments inside code cells. They are ignored in markdown cells.

General form:

# pytest-nb-as-test: <flag>=<value>

Rules:

• each flag may appear at most once per cell
• booleans accept True or False (case sensitive)
• timeouts accept numeric seconds
• invalid values, or repeated flags, fail at collection time

default-all

Sets the default inclusion status for subsequent code cells.

# pytest-nb-as-test: default-all=True|False

Example:

# pytest-nb-as-test: default-all=False
# cells from here are skipped

# ... plotting, exploration, notes ...

# pytest-nb-as-test: default-all=True
# execution resumes

test-cell

Overrides the current default for the current cell only.

# pytest-nb-as-test: test-cell=True|False

must-raise-exception

Marks a cell as expected to raise an exception.

# pytest-nb-as-test: must-raise-exception=True|False

If True, the cell is executed under pytest.raises(Exception). The test fails if no exception is raised, or if a BaseException (for example SystemExit) is raised.

Example:

# pytest-nb-as-test: must-raise-exception=True
raise ValueError("Intentional failure for demonstration")

notebook-timeout-seconds

Sets a wall clock timeout (seconds) for the whole notebook. Requires pytest-timeout. Must appear in the first code cell.

# pytest-nb-as-test: notebook-timeout-seconds=<float>

cell-timeout-seconds

Sets a per cell timeout (seconds). Requires pytest-timeout.

# pytest-nb-as-test: cell-timeout-seconds=<float>

Configuration

Precedence order:

1. In notebook directives
2. CLI options when explicitly provided
3. pytest.ini or pyproject.toml
4. defaults

This plugin does not currently read environment variables for configuration.

CLI options

Option	Type	Default	Description
--notebook-default-all	true false	true	Initial value of the test_all_cells flag. If false then cells without an explicit test-cell directive will be skipped until default-all=True is encountered.
--notebook-glob	string	none	Glob pattern for notebook filenames, name-only patterns match basenames, path patterns match relative paths.
--notebook-keep-generated	none onfail <path>	onfail	Controls dumping of the generated test script. none means never dump, onfail dumps the script into the report upon a test failure, any other string is treated as a path and the script is written there with a filename derived from the notebook name.
--notebook-exec-mode	auto async sync	auto	Execution mode for the wrapper function. auto (default) auto-detects await statements and generates async def only when needed; async forces async def regardless; sync forces synchronous execution. Async code is executed with asyncio.run().
--notebook-timeout-seconds	float	none	Wall-clock timeout for an entire notebook, enforced via pytest-timeout.
--notebook-cell-timeout-seconds	float	none	Default per-cell timeout in seconds, enforced via pytest-timeout.

pytest.ini / pyproject.toml settings

You can set options in your pytest.ini or pyproject.toml under [tool.pytest.ini_options]. In ini files, use the underscore option names (notebook_default_all), not the CLI flag form with dashes. For example:

[pytest]
notebook_default_all = false
notebook_timeout_seconds = 120
notebook_cell_timeout_seconds = 10
notebook_glob = test_*.ipynb

Values set in the ini file are overridden by CLI flags that you pass explicitly.

In pyproject.toml, put the same keys under [tool.pytest.ini_options].

Note: notebook_default_all = false only changes which cells are selected inside notebooks; it does not disable notebook collection. To skip notebook tests entirely, use pytest selection options like -m "not notebook" (marker expression; this plugin marks notebook items with notebook) or --ignore-glob=*.ipynb (pytest built-in) in addopts.

Example (CLI):

pytest -m "not notebook"

Example (pytest.ini):

[pytest]
addopts = -m "not notebook"

Debugging failures

On failure, the plugin can attach the generated Python script to the pytest report. With --notebook-keep-generated=onfail (default) you get a “generated notebook script” section in the report.

If you pass a directory to --notebook-keep-generated, the script is written there with a name derived from the notebook filename.

Each selected cell is preceded by a marker comment:

## pytest-nb-as-test notebook=<filename> cell=<index>

Use this to correlate tracebacks with notebook cell indices.

Versioning / API stability

This project follows Semantic Versioning.

Before 1.0, public APIs may change without notice. After 1.0, the following are considered stable public APIs:

• CLI options listed in this README.
• pytest.ini / pyproject.toml configuration keys listed in this README.
• Notebook directives (default-all, test-cell, must-raise-exception, notebook-timeout-seconds, cell-timeout-seconds).

Behavioral changes to these APIs will be announced in the changelog and, when practical, introduced with a deprecation period of at least one minor release.

Demo

Run the demo harness:

python run_demo.py

It copies a small set of notebooks into a temporary workspace, invokes pytest, and reports outcomes.

Development and testing

The plugin tests live in tests/test_plugin.py and use notebooks under tests/notebooks/.

Run:

pytest

Examples:

pytest tests/notebooks/example_simple_123.ipynb
pytest tests/notebooks --notebook-glob "test_*.ipynb"

Suggested conftest snippets

Put these in a conftest.py near your notebooks and keep them scoped to notebook tests via the notebook marker.

NumPy RNG: seed and ensure it is unused

import pytest


@pytest.fixture(autouse=True)
def seed_and_lock_numpy_rng(request: pytest.FixtureRequest) -> None:
    if request.node.get_closest_marker("notebook") is None:
        yield
        return

    try:
        import numpy as np
    except ModuleNotFoundError:
        yield
        return

    np.random.seed(0)
    state = np.random.get_state()
    yield
    new_state = np.random.get_state()

    same_state = (
        state[0] == new_state[0]
        and state[2:] == new_state[2:]
        and np.array_equal(state[1], new_state[1])
    )
    if not same_state:
        raise AssertionError("NumPy RNG state changed; random was called.")

Matplotlib backend

import pytest


@pytest.fixture(autouse=True)
def set_matplotlib_backend(request: pytest.FixtureRequest) -> None:
    if request.node.get_closest_marker("notebook") is None:
        yield
        return

    try:
        import matplotlib
    except ModuleNotFoundError:
        yield
        return

    matplotlib.use("Agg")
    yield

Plotly renderer

import pytest


@pytest.fixture(autouse=True)
def set_plotly_renderer(request: pytest.FixtureRequest) -> None:
    if request.node.get_closest_marker("notebook") is None:
        yield
        return

    try:
        import plotly.io as pio
    except ModuleNotFoundError:
        yield
        return

    os.environ.setdefault("PLOTLY_RENDERER", "json")

    import plotly.io as pio

    pio.renderers.default = "json"
    pio.renderers.render_on_display = False
    pio.show = lambda *args, **kwargs: None
    yield

Prior art and related tools

Several existing projects test notebooks, but they optimise for different goals.

Output regression testing (compare stored outputs)

• nbval: collects notebooks, executes them in a Jupyter kernel, and compares executed cell outputs against the outputs stored in the .ipynb (each cell behaves like a test). It also supports output sanitisation for noisy outputs.
  https://pypi.org/project/nbval/
• pytest-notebook: executes notebooks, diffs input vs output notebooks (via nbdime), and can regenerate notebooks when outputs change. Also integrates with coverage tooling.
  https://pytest-notebook.readthedocs.io/ When to prefer these: you want to detect changes in rendered outputs, not just “runs without error”.

Execute notebooks under pytest (smoke execution, not output diffs)

• pytest-nbmake: executes notebooks during pytest using nbclient. Supports per-cell behaviour via notebook cell tags (for example skip-execution, raises-exception).
  https://github.com/treebeardtech/pytest-nbmake When to prefer this: you want faithful notebook execution semantics (kernel based execution) and simple CI integration.

“Tests inside notebooks” (interactive and teaching workflows)

• pytest-ipynb2: collects tests written in notebooks via a %%ipytest magic, supports fixtures and parametrisation, and executes cells above the test cell.
  https://musicalninjadad.github.io/pytest-ipynb2/
• ipytest: run pytest conveniently from within a notebook (primarily interactive UX).
  https://github.com/chmp/ipytest
• nbtest-plugin: provides notebook-friendly assertion helpers (including DataFrame assertions) that are later collected by pytest when run with --nbtest.
  https://pypi.org/project/nbtest-plugin/
• nbcelltests: cell-by-cell testing aimed at “linearly executed notebooks”, with JupyterLab integration.
  https://github.com/jpmorganchase/nbcelltests It integrates with JupyterLab via bundled lab and server extensions, so tests can be authored and run from the browser. Tests are stored in cell metadata, and nbcelltests generates a Python unittest class with per cell methods whose state includes the cumulative context of all prior cells, mimicking linear execution. Inside a test you can use %cell to inject the corresponding notebook cell source into the generated test method. It can also run offline from an .ipynb, and it supports a lint mode plus additional structural checks such as maximum lines per cell, maximum cells per notebook, maximum number of function or class definitions, and minimum percentage of cells tested.

How pytest-nb-as-test differs

This plugin is aimed at CI enforcement of example notebooks in scientific codebases, with two deliberate design choices:

1. In-process execution so that normal pytest mechanisms (fixtures, monkeypatch, markers) can apply to notebook code.
2. Per-cell directives embedded in code cell comments (default-all, test-cell, timeouts, expected exceptions), so behaviour is visible in diffs without relying on notebook metadata.

If you need output regression diffs, prefer nbval or pytest-notebook. If you need faithful kernel execution semantics, prefer pytest-nbmake.