Skip to main content

Basic notebook checks. Do they run? Do they contain lint?

Project description


Build Status Linux/MacOS/Windows Build Status
Coverage codecov
Latest dev release Github tag
Latest release Github release PyPI version nbsmoke version conda-forge version defaults version
Python Python support

Basic notebook smoke tests: Do they run ok? Do they contain lint?

This Pytest plugin was generated with Cookiecutter along with @hackebrot's Cookiecutter-pytest-plugin template.


You can install nbsmoke via pip from PyPI:

$ pip install nbsmoke

Or get the latest pre-release:

$ pip install --pre nbsmoke

nbsmoke is also available via conda from

$ conda install -c conda-forge nbsmoke


Check all notebooks run without errors:

$ pytest --nbsmoke-run

Check all notebooks run without errors, and store html to look at afterwards:

$ pytest --nbsmoke-run --store-html=/scratch

Lint check notebooks:

$ pytest --nbsmoke-lint

Lint failures as warnings only:

$ pytest --nbsmoke-lint --nbsmoke-lint-onlywarn

Instead of all files in a directory, you can specify a list e.g.:

$ pytest --nbsmoke-run notebooks/Untitled*.ipynb

If you want to restrict pytest to running only your notebook tests, use -k, e.g.:

$ pytest --nbsmoke-run -k ".ipynb"

Additional options are available by standard pytest 'ini' configuration in setup.cfg, pytest.ini, or tox.ini:

# when running, seconds allowed per cell (see nbconvert timeout)
nbsmoke_cell_timeout = 600

# notebooks to skip running; one case insensitive re to match per line
nbsmoke_skip_run = ^.*skipme\.ipynb$

# case insensitive re to match for file to be considered notebook;
# defaults to ``^.*\.ipynb``
it_is_nb_file = ^.*\.something$

# flakes you don't want to hear about (regex)
nbsmoke_flakes_to_ignore = .*hvplot.* imported but unused.*

# line magics to treat as being flakes (i.e. magics you don't want in your notebooks)
nbsmoke_flakes_line_magics_blacklist = pylab

# cell magics to treat as being flakes (i.e. magics you don't want in your notebooks)
nbsmoke_flakes_cell_magics_blacklist = bash

# add your own magic handlers (python file containing line_magic_handlers and cell_magic_handlers as dictionaries magic_name: callable)
nbsmoke_magic_handlers = path/to/

nbsmoke supports # noqa comments to mark that something should be ignored during lint checking.

The nbsmoke_skip_run list in a project's config can be ignored by passing --ignore-nbsmoke-skip-run (useful if sometimes you want to run all notebooks for a project where many are typically skipped).

What's the point?

Although more sophisticated testing of notebooks is possible (e.g. see nbval), just checking that notebooks run from start to finish without error in a fresh kernel (or on a neutral CI service) can be useful during development. Practical experience of working on several projects with notebooks confirms this, but that's all the evidence I have.

Checking notebooks for lint might seem trivial/pointless, but it frequently uncovers unused names (typically unused imports). It's also quite common to find python 2 vs 3 problems, and sometimes undefined names - in a way that's faster than running the notebook (over multiple versions of python).

Unused imports/names themselves might seem trivial, but they can hinder understanding of a notebook by readers, or add dependencies that are not required.

Hopefully you don't have mysterious (unused) imports in your notebook, but if you do, you can add # noqa: explanation to stop flake errors. E.g. if you're importing something for its side effects, it's very helpful to inform the reader of that.

Pyflakes is used as the underlying linter because "Pyflakes makes a simple promise: it will never complain about style, and it will try very, very hard to never emit false positives."


First, install using pip install -e .. Then run the tests using tox or pytest -v tests/.

New release to PyPI: git tag -a vX.Y.Z -m "Something about release" && git push --tags. Then a PR will auto-open on conda-forge, which should be merged.

Get some help to debug apparently incorrect flakes by adding --nbsmoke-lint-debug, e.g. pytest -v --nbsmoke-lint --nbsmoke-lint-debug examples.


Distributed under the terms of the BSD-3 license, "nbsmoke" is free and open source software.


If you encounter any problems, please file an issue (ideally including a copy of any problematic notebook).

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nbsmoke-0.6.0.tar.gz (25.2 kB view hashes)

Uploaded Source

Built Distribution

nbsmoke-0.6.0-py2.py3-none-any.whl (28.2 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page