Skip to main content

pytest-nb-as-test Plugin

CI pipeline status

icon

In scientific codebases, notebooks are a convenient way to provide executable examples, figures, and LaTeX. However, example notebooks often become silently broken as the code evolves because developers rarely re-run them. New users then discover the breakage when they try the examples, which is disheartening and frustrating. This plugin executes notebook code cells as pytest tests, so example notebooks run in CI and stay up to date.

When to use

  • You want .ipynb notebooks collected by pytest and run in CI.
  • You want in process execution, so fixtures and monkeypatching apply.
  • You need per cell control (skip, force run, expect exception, timeouts) via directives.

For comparison with other plugins/ projects see Prior art and related tools.

Install

install using pip

pip install pytest-nb-as-test

or add as dependency in pyproject.toml:

[project]
dependencies = [
  "pytest-nb-as-test",
]

Run

Pytest discovers all notebooks alongside normal tests:

pytest

Filter which notebooks are collected:

pytest --notebook-glob 'test_*.ipynb'

Disable notebook collection and execution:

pytest -p no:pytest_nb_as_test

IPython Runtime Compatibility

pytest-nb-as-test executes transformed Python code in-process under pytest. It does not execute notebooks through a live IPython/Jupyter kernel.

Current contract:

  • Lines that start with %, %%, or ! are commented out before execution.
  • get_ipython() and IPython globals (In, Out, _ih, _oh, _dh) are not provided by the runtime.
  • from __future__ import ... lines are hoisted to the top of generated code across selected cells.

When unsupported IPython runtime usage is detected, and when future imports appear outside the first selected code cell, the plugin emits a PytestWarning during collection so it is visible in pytest output.

Implication: a notebook can pass while skipping notebook-only side effects from magics or shell escapes. Also, future import behavior differs from Jupyter: in Jupyter, a future import only affects cells compiled after that cell executes; in pytest-nb-as-test, hoisted future imports apply globally to selected cells in the generated script. When a future import appears outside the first selected code cell, the plugin emits a PytestWarning during collection. If you need strict kernel-faithful semantics, use a kernel-based plugin such as pytest-nbmake.

Cell directives

Directives live in comments inside code cells. They are ignored in markdown cells.

General form:

# pytest-nb-as-test: <flag>=<value>

Rules:

  • each flag may appear at most once per cell
  • booleans accept True or False (case sensitive)
  • timeouts accept numeric seconds
  • invalid values, or repeated flags, fail at collection time

default-all

Sets the default inclusion status for subsequent code cells.

# pytest-nb-as-test: default-all=True|False

Example:

# pytest-nb-as-test: default-all=False
# cells from here are skipped

# ... plotting, exploration, notes ...

# pytest-nb-as-test: default-all=True
# execution resumes

test-cell

Overrides the current default for the current cell only.

# pytest-nb-as-test: test-cell=True|False

must-raise-exception

Marks a cell as expected to raise an exception.

# pytest-nb-as-test: must-raise-exception=True|False

If True, the cell is executed under pytest.raises(Exception). The test fails if no exception is raised, or if a BaseException (for example SystemExit) is raised.

Example:

# pytest-nb-as-test: must-raise-exception=True
raise ValueError("Intentional failure for demonstration")

notebook-timeout-seconds

Sets a wall clock timeout (seconds) for the whole notebook. Requires pytest-timeout. Must appear in the first code cell.

# pytest-nb-as-test: notebook-timeout-seconds=<float>

cell-timeout-seconds

Sets a per cell timeout (seconds). Requires pytest-timeout.

# pytest-nb-as-test: cell-timeout-seconds=<float>

Configuration

Precedence order:

  1. In notebook directives
  2. CLI options when explicitly provided
  3. pytest.ini or pyproject.toml
  4. defaults

This plugin does not currently read environment variables for configuration.

CLI options

Option Type Default Description
--notebook-default-all true false true Initial value of the test_all_cells flag. If false then cells without an explicit test-cell directive will be skipped until default-all=True is encountered.
--notebook-glob string none Glob pattern for notebook filenames, name-only patterns match basenames, path patterns match relative paths.
--notebook-keep-generated none onfail <path> onfail Controls dumping of the generated test script. none means never dump, onfail dumps the script into the report upon a test failure, any other string is treated as a path and the script is written there with a filename derived from the notebook name.
--notebook-exec-mode auto async sync auto Execution mode for the wrapper function. auto (default) auto-detects await statements and generates async def only when needed; async forces async def regardless; sync forces synchronous execution. Async code is executed with asyncio.run().
--notebook-timeout-seconds float none Wall-clock timeout for an entire notebook, enforced via pytest-timeout.
--notebook-cell-timeout-seconds float none Default per-cell timeout in seconds, enforced via pytest-timeout.

pytest.ini / pyproject.toml settings

You can set options in your pytest.ini or pyproject.toml under [tool.pytest.ini_options]. In ini files, use the underscore option names (notebook_default_all), not the CLI flag form with dashes. For example:

[pytest]
notebook_default_all = false
notebook_timeout_seconds = 120
notebook_cell_timeout_seconds = 10
notebook_glob = test_*.ipynb

Values set in the ini file are overridden by CLI flags that you pass explicitly.

In pyproject.toml, put the same keys under [tool.pytest.ini_options].

Note: notebook_default_all = false only changes which cells are selected inside notebooks; it does not disable notebook collection. To skip notebook tests entirely, use pytest selection options like -m "not notebook" (marker expression; this plugin marks notebook items with notebook) or --ignore-glob=*.ipynb (pytest built-in) in addopts.

Example (CLI):

pytest -m "not notebook"

Example (pytest.ini):

[pytest]
addopts = -m "not notebook"

Debugging failures

On failure, the plugin can attach the generated Python script to the pytest report. With --notebook-keep-generated=onfail (default) you get a “generated notebook script” section in the report.

If you pass a directory to --notebook-keep-generated, the script is written there with a name derived from the notebook filename.

Each selected cell is preceded by a marker comment:

## pytest-nb-as-test notebook=<filename> cell=<index>

Use this to correlate tracebacks with notebook cell indices. Here <index> is the notebook JSON cell index (zero-based, includes markdown/raw cells).

Multiprocessing scope

Goal: multiprocessing behavior should match what works in a normal Jupyter notebook workflow, not exceed it.

In practice, this means prioritising parity for notebook patterns that Jupyter kernels handle successfully (for example Linux fork with top-level notebook definitions), and not targeting broader support for patterns that are typically not reliable in Jupyter (for example spawn/forkserver importability edge cases, lambdas, nested callables).

Versioning / API stability

This project follows Semantic Versioning.

Before 1.0, public APIs may change without notice. After 1.0, the following are considered stable public APIs:

  • CLI options listed in this README.
  • pytest.ini / pyproject.toml configuration keys listed in this README.
  • Notebook directives (default-all, test-cell, must-raise-exception, notebook-timeout-seconds, cell-timeout-seconds).

Behavioral changes to these APIs will be announced in the changelog and, when practical, introduced with a deprecation period of at least one minor release.

Demo

Run the demo harness:

python run_demo.py

It copies a small set of notebooks into a temporary workspace, invokes pytest, and reports outcomes.

Development and testing

The plugin tests live in tests/test_plugin.py and use notebooks under tests/notebooks/.

Run:

pytest

Examples:

pytest tests/notebooks/example_simple_123.ipynb
pytest tests/notebooks --notebook-glob "test_*.ipynb"

Suggested conftest snippets

Put these in a conftest.py near your notebooks and keep them scoped to notebook tests via the notebook marker.

NumPy RNG: seed and ensure it is unused

from collections.abc import Generator

import pytest


@pytest.fixture(autouse=True)
def seed_and_lock_numpy_rng(
    request: pytest.FixtureRequest,
) -> Generator[None, None, None]:
    if request.node.get_closest_marker("notebook") is None:
        yield
        return

    try:
        import numpy as np
    except ModuleNotFoundError:
        yield
        return

    np.random.seed(0)
    state = np.random.get_state()
    yield
    new_state = np.random.get_state()

    same_state = (
        state[0] == new_state[0]
        and state[2:] == new_state[2:]
        and np.array_equal(state[1], new_state[1])
    )
    if not same_state:
        raise AssertionError("NumPy RNG state changed; random was called.")

Matplotlib backend

from collections.abc import Generator

import pytest


@pytest.fixture(autouse=True)
def set_matplotlib_backend(
    request: pytest.FixtureRequest,
) -> Generator[None, None, None]:
    if request.node.get_closest_marker("notebook") is None:
        yield
        return

    try:
        import matplotlib
    except ModuleNotFoundError:
        yield
        return

    matplotlib.use("Agg")
    yield

Plotly renderer

import os
from collections.abc import Generator

import pytest


@pytest.fixture(autouse=True)
def set_plotly_renderer(
    request: pytest.FixtureRequest,
) -> Generator[None, None, None]:
    if request.node.get_closest_marker("notebook") is None:
        yield
        return

    try:
        import plotly.io as pio
    except ModuleNotFoundError:
        yield
        return

    os.environ.setdefault("PLOTLY_RENDERER", "json")

    pio.renderers.default = "json"
    pio.renderers.render_on_display = False
    pio.show = lambda *args, **kwargs: None
    yield

Several existing projects test notebooks, but they optimise for different goals.

Output regression testing (compare stored outputs)

  • nbval: collects notebooks, executes them in a Jupyter kernel, and compares executed cell outputs against the outputs stored in the .ipynb (each cell behaves like a test). It also supports output sanitisation for noisy outputs.
    https://pypi.org/project/nbval/
  • pytest-notebook: executes notebooks, diffs input vs output notebooks (via nbdime), and can regenerate notebooks when outputs change. Also integrates with coverage tooling.
    https://pytest-notebook.readthedocs.io/ When to prefer these: you want to detect changes in rendered outputs, not just “runs without error”.

Execute notebooks under pytest (smoke execution, not output diffs)

  • pytest-nbmake: executes notebooks during pytest using nbclient. Supports per-cell behaviour via notebook cell tags (for example skip-execution, raises-exception).
    https://github.com/treebeardtech/pytest-nbmake When to prefer this: you want faithful notebook execution semantics (kernel based execution) and simple CI integration.

“Tests inside notebooks” (interactive and teaching workflows)

  • pytest-ipynb2: collects tests written in notebooks via a %%ipytest magic, supports fixtures and parametrisation, and executes cells above the test cell.
    https://musicalninjadad.github.io/pytest-ipynb2/
  • ipytest: run pytest conveniently from within a notebook (primarily interactive UX).
    https://github.com/chmp/ipytest
  • nbtest-plugin: provides notebook-friendly assertion helpers (including DataFrame assertions) that are later collected by pytest when run with --nbtest.
    https://pypi.org/project/nbtest-plugin/
  • nbcelltests: cell-by-cell testing aimed at “linearly executed notebooks”, with JupyterLab integration.
    https://github.com/jpmorganchase/nbcelltests It integrates with JupyterLab via bundled lab and server extensions, so tests can be authored and run from the browser. Tests are stored in cell metadata, and nbcelltests generates a Python unittest class with per cell methods whose state includes the cumulative context of all prior cells, mimicking linear execution. Inside a test you can use %cell to inject the corresponding notebook cell source into the generated test method. It can also run offline from an .ipynb, and it supports a lint mode plus additional structural checks such as maximum lines per cell, maximum cells per notebook, maximum number of function or class definitions, and minimum percentage of cells tested.

How pytest-nb-as-test differs

This plugin is aimed at CI enforcement of example notebooks in scientific codebases, with two deliberate design choices:

  1. In-process execution so that normal pytest mechanisms (fixtures, monkeypatch, markers) can apply to notebook code.
  2. Per-cell directives embedded in code cell comments (default-all, test-cell, timeouts, expected exceptions), so behaviour is visible in diffs without relying on notebook metadata.

If you need output regression diffs, prefer nbval or pytest-notebook. If you need faithful kernel execution semantics, prefer pytest-nbmake.

Planned

Stricter must-raise-exception matching (target: v1.1)

v1.0 supports a boolean must-raise-exception=True|False. When enabled, any Exception satisfies the directive.

Planned enhancement:

  • allow matching the expected exception type separately from the message
  • keep current boolean behavior as the default for backward compatibility

Proposed directives:

# pytest-nb-as-test: must-raise-exception=True
# pytest-nb-as-test: must-raise-exception-type=ValueError
# pytest-nb-as-test: must-raise-exception-match=intentional error

Intended behavior:

  • must-raise-exception-type validates the exception class
  • must-raise-exception-match validates the error message (regex-style match)
  • both are optional refinements when must-raise-exception=True

Metadata

Release files for pytest-nb-as-test 1.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pytest-nb-as-test 1.0.1
File Size Uploaded Provenance Actions
pytest_nb_as_test-1.0.1.tar.gz 31.2 kB With provenance Details for pytest_nb_as_test-1.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pytest-nb-as-test 1.0.1
File Interpreter ABI Platform Actions
pytest_nb_as_test-1.0.1-py3-none-any.whl Python 3 none any Details for pytest_nb_as_test-1.0.1-py3-none-any.whl

Total release size: 59.2 kB

Release files / pytest_nb_as_test-1.0.1.tar.gz

Download URL pytest_nb_as_test-1.0.1.tar.gz
Size 31.2 kB
Tags Source
SHA-256 checksum
How to use checksums
0d4fe4e5c8881d0d41d1f9f8dcab00f142dfb4c067c9df24e6a324e72159c6be
BLAKE2b-256 checksum
How to use checksums
8dfaa139513d3dc3501d52832fdc554ee5d54bc3eeade88a54463fe3958c4933
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 15, 2026.

Transparency log

Release files / pytest_nb_as_test-1.0.1-py3-none-any.whl

Download URL pytest_nb_as_test-1.0.1-py3-none-any.whl
Size 28.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
cf63b1cec23a8c2c6522859c70f507e04b6ba53b7fbd2d10b9f14607c3f7e3d9
BLAKE2b-256 checksum
How to use checksums
864f6cf853424f8f15e142d296db2c937f75f2e7b24fca362fdb3cc094d8843d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 15, 2026.

Transparency log

Security for pytest-nb-as-test 1.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pytest_nb_as_test-1.0.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 15, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page