Skip to main content

nbwipers

Test License:MIT PyPI - Version Crates.io Conda codecov

nbwipers is a command line tool to wipe clean jupyter notebooks, written in Rust.

The interface and functionality are based on nbstripout and the idea to implement it in rust comes from nbstripout-fast.

Usage

nbwipers has a few subcommands that provide functionality related to cleaning Jupyter notebooks.

  • clean: clean a single notebook. This is more-or-less equivalent to nbstripout.
  • check: check notebooks in a given path for elements that would be removed by clean. This could be used in a CI context to enforce clean notebooks.
  • clean-all clean all notebooks in a given path. This one should be used carefully!
  • install register nbwipers as a git filter for ipynb files. Equivalent to nbstripout --install
  • uninstall remove nbwipers as a git filter.
  • check-install check that nbwipers or nbstripout is installed in the local repo. This is used in the pre-commit hook.
  • show-config show the effective configuration nbwipers would use, merging the config file with any CLI overrides.
  • record record kernel metadata for notebooks in a local, git-untracked store, so it can be restored later even though strip-kernel-info removes it from committed notebooks. See Preserving kernel info locally below.
  • hook subcommands used by pre-commit-style hooks — currently check-large-files, which checks notebook file sizes after cleaning.

The full options can be found in CommandLineHelp.md.

Examples

To set up nbwipers as a git filter in your repository, use

nbwipers install local

If this step is performed on a pre-existing repo, you can touch your notebooks so that git can detect the changes. In bash:

for f in $(git ls-files '*.ipynb'); do touch $f; done

To check the notebooks in your folder, you can run the following

nbwipers check .

To see the configuration nbwipers would use in the current directory, you can run

nbwipers show-config

Add --show-all to also see the default values for settings that have not been explicitly configured.

Preserving kernel info locally

nbwipers install sets up both a clean filter, which strips notebooks before they are committed, and a smudge filter, which runs when notebooks are checked out.

If you enable strip-kernel-info (see Configuration) so that kernelspec and python-version metadata never gets committed, you can still keep that information around locally with record:

nbwipers record .

This saves the kernel metadata for notebooks under the given path to .git/x-nbwipers/kernelspec_store.json — local to your clone and never committed. The next time you check out one of those notebooks, the smudge filter automatically restores its recorded kernel metadata, so each collaborator keeps their own kernel/python version info without it living in version control.

To keep the local store tidy as notebooks come and go:

  • nbwipers record --sync . discards the whole store and rebuilds it from the notebooks currently found under ., dropping entries for notebooks that no longer exist.
  • nbwipers record --remove path/to/notebook.ipynb removes a specific notebook's entry, leaving the rest of the store untouched.
  • nbwipers record --clear wipes the store entirely.

pre-commit

You can add the following to your pre-commit-config.yaml file to ensure that nbwipers or nbstripout is installed in your repo, in order to prevent Jupyter notebook outputs from being committed to version control.

  - repo: https://github.com/felixgwilliams/nbwipers-pre-commit
    rev: v0.7.1
    hooks:
      - id: nbwipers-check-install

Alternatively, you can use the URL for this repo in your config, but this will compile nbwipers from source, rather than retrieving the binary from PyPI, and is therefore not recommended.

If you are using your pre-commit configuration as part of CI, you should set the environment variable NBWIPERS_CHECK_INSTALL_EXIT_ZERO which forces this check to pass, since you do not need nbwipers configured in your CI environment.

Configuration

Configuration is currently done via the tool.nbwipers section of the pyproject.toml file. Most of the command line options can be set per-project in the pyproject.toml, nbwipers.toml or .nbwipers.toml file. If you use pyroject.toml, you need to put the configuration under [tool.nbwipers]. If you use nbwipers.toml or .nbwipers.toml, the configuration needs to be at the top level.

For example you can use strip-kernel-info to remove metadata on the python version or the details about the Jupyter Kernel.

You can also drop cell ids using id-action = "drop".

To enable these options, you can include the following in your pyproject.toml file:

[tool.nbwipers]
strip-kernel-info = true
id-action = "drop"

The equivalent for nbwipers.toml or .nbwipers.toml is just

strip-kernel-info = true
id-action = "drop"

This can be useful when collaborating, as the precise python version and the name assigned to the kernel are ephemeral and can change from person to person. Cell IDs are another element of the file which is generated by the tool you use and can change from person to person.

Motivation

A working copy of a Jupyter notebook contains:

  1. Code written by the author.
  2. Notebook outputs: tables, logs, tracebacks, images, widgets and so on...
  3. Execution counts.
  4. Metadata, such as whether cells are collapsed, scrollable etc.

Of these categories of data, only the first — code written by the author — should definitely be tracked by version control, since it is the product of the author's intention and hard work. The other categories of data are subject to change outside of the explicit intentions of the author and are generally noisy from a version control perspective.

Moreover, including notebook outputs in version control

  • makes diffs harder to interpret, as they will contain lots of unintended changes.
  • increases the risk of a tricky merge conflict if different users run the same cell and get a slightly different result.
  • increases the amount of data committed, which can degrade repository performance.
  • risks leaking sensitive data.

An effective way to ensure you do not commit problematic parts of your notebooks is to use nbwipers or nbstripout as a git filter.

A git filter sits between your actual files and what git sees when you stage and commit your changes. This way, git only sees the transformed version of the file without the problematic elements. At the same time, you do not have to lose them from your local copy.

An exception is when you checkout a branch or do a git pull, which results in changes to the notebook. In this case, your local copy will be replaced by the clean version and you will lose your cell outputs.

Acknowledgements

nbwipers relies on inspiration and code from several projects. For the projects, whose code was used please see LICENSE for the third-party notices.

nbstripout

strip output from Jupyter and IPython notebooks

nbstripout is an invaluable tool for working with Jupyter Notebooks in the context of version control. This project forms the basis of the interface and logic of this project and is also the source of the testing examples.

nbstripout-fast

A much faster version of nbstripout by writing it in rust (of course).

nbstripout-fast, like this project, implements the functionality of nbstripout in Rust, while also allowing repo-level configuration in a YAML file.

With nbwipers I hoped to recreate the idea of nbstripout-fast, but with the ability to install as a git filter, and configuration via pyproject.toml.

ruff

An extremely fast Python linter and code formatter, written in Rust.

Ruff is quickly becoming the linter for python code, thanks to its performance, extensive set of rules and its ease of use. It was a definite source of knowledge for the organisation of the configuration and the file discovery. The schema for Jupyter Notebooks, and some of the file discovery code was adapted from Ruff.

pre-commit

A framework for managing and maintaining multi-language pre-commit hooks.

This repo contains a version of the check-large-files hook, that will not flag notebook files whose clean size is less that the threshold, even if the size on-disk including outputs is greater than the threshold. The logic and interface of the hook was adapted from the pre-commit-hooks repository.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nbwipers-0.7.1.tar.gz (78.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

nbwipers-0.7.1-py3-none-win_amd64.whl (1.9 MB view details)

Uploaded Python 3Windows x86-64

nbwipers-0.7.1-py3-none-win32.whl (1.7 MB view details)

Uploaded Python 3Windows x86

nbwipers-0.7.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

nbwipers-0.7.1-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl (2.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ s390x

nbwipers-0.7.1-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (2.6 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ppc64le

nbwipers-0.7.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl (2.4 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ i686

nbwipers-0.7.1-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (2.1 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARMv7l

nbwipers-0.7.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.2 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

nbwipers-0.7.1-py3-none-macosx_11_0_arm64.whl (2.0 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

nbwipers-0.7.1-py3-none-macosx_10_12_x86_64.whl (2.1 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

File details

Details for the file nbwipers-0.7.1.tar.gz.

File metadata

  • Download URL: nbwipers-0.7.1.tar.gz
  • Upload date:
  • Size: 78.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for nbwipers-0.7.1.tar.gz
Algorithm Hash digest
SHA256 b742331a8a6570d99bb4afa74bac8864a6102fb8c1a89017fc607be3a50353e3
MD5 2f8a0f29c44cc7c7cc36a4fdf355d660
BLAKE2b-256 0c19409012b1a4abe0c5e8d930c0c9dd413c1ed3bd80e44d5b2547d3eab257ac

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1.tar.gz:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbwipers-0.7.1-py3-none-win_amd64.whl.

File metadata

  • Download URL: nbwipers-0.7.1-py3-none-win_amd64.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: Python 3, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for nbwipers-0.7.1-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 b16e6034691d59dd4dccbc40fe8fa0cf5dc1ddab686e5eb03a41dfda2b1fab88
MD5 de6289292c03024a20427a5549efe799
BLAKE2b-256 48c1c06cab71ada9ab424e9aebc12fd6577cfe7cd53281cd8ec7f6a6ce9bfb81

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1-py3-none-win_amd64.whl:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbwipers-0.7.1-py3-none-win32.whl.

File metadata

  • Download URL: nbwipers-0.7.1-py3-none-win32.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: Python 3, Windows x86
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for nbwipers-0.7.1-py3-none-win32.whl
Algorithm Hash digest
SHA256 cf3e247031c118bc7c88f2bea8eac339f6417d4ce4c1bbde1fd2ab96a3a0db2c
MD5 4631aaf8f62fb8155c9bf6876d32643d
BLAKE2b-256 a4fbed0abd491f48df0289c9842a7b12f0580556dfc67388e1692fed409072ef

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1-py3-none-win32.whl:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbwipers-0.7.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for nbwipers-0.7.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 81b4844b0f8d3f2590ea745ac6f1a1608339d05e1578a23284edf124d338da43
MD5 f5c0678b6fea5fdb333dbd3e3305e394
BLAKE2b-256 2deca31caff0a43f9b96de57c8578bf0802e506c4aa7ce91e1a83e0c12e76064

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbwipers-0.7.1-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

File hashes

Hashes for nbwipers-0.7.1-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 ec633890f92e2cb50ba3d5fab7175427523f2ce7fd66717dbbab544e63e4fe20
MD5 d641509c58997dddfa80a2bab393f81f
BLAKE2b-256 355a54b5eb6eb33d536d18e5a1d06048a436ee546e70a3c578b1288765933b0b

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbwipers-0.7.1-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

File hashes

Hashes for nbwipers-0.7.1-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 581b3a98c068a0d87f626b0d0dfd04b0d0fd0e560a857620833e18cf83e33511
MD5 17cf26814efd11f4fe54646f4d644a24
BLAKE2b-256 3fa25019e61ebcb7d87a59206a02d5cfa028e427f2f644e84588011362d57067

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbwipers-0.7.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for nbwipers-0.7.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 d7374c869988c6b0f61d0338894bb33289922f35dcd7af5197ea2ccd4a3ae2ed
MD5 cf4ce5aa0e72e601d903583be47410b2
BLAKE2b-256 5b2894bf1a58b44e77e2667792392b10d2a5d4ce21a0ffd692a1926b6cc4aa6d

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbwipers-0.7.1-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl.

File metadata

File hashes

Hashes for nbwipers-0.7.1-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm Hash digest
SHA256 271ee79d2e8366f5f908ee07e21108608338464b5a96451b483198357fbe91ce
MD5 dc3824d8aaf7a066241ef0b8283e1a5c
BLAKE2b-256 78d0fec07d5cd5c7b820739a69abd781c624422c10ac17a50353dfb8950cff66

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbwipers-0.7.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for nbwipers-0.7.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 4463a949cb0b0de2ef0bd07948a1c06aacf08b96cee09e07450619aef02b5997
MD5 a90a49b638a88d98e7de45b066fb89bb
BLAKE2b-256 5800af0b67a13a8df3073293f8ececaf64e7cc1824029a589a829b2a06529800

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbwipers-0.7.1-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for nbwipers-0.7.1-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 ecdd1a37e26395d6358661a3363ef1b5301bbf647d7816c29a01db1fc65fb975
MD5 1625f1bb86f9f1d7afb3ef3d405d8213
BLAKE2b-256 f2a4f2833dfe4b3a4efca1bc33d64e762293b0df23e638b40d4529bc085c6b15

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1-py3-none-macosx_11_0_arm64.whl:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbwipers-0.7.1-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for nbwipers-0.7.1-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 6e07798a993a7cc54d5cde0d34e81a8904d7f5bb5228e616a83581d9f997ef20
MD5 ad2b1879ff52179e134cb02aaf3d4135
BLAKE2b-256 b5a8a86e62eaaedc0e3eb367405ffddf0798e3d9aedcba28337a6da9246e44d5

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbwipers-0.7.1-py3-none-macosx_10_12_x86_64.whl:

Publisher: release.yml on felixgwilliams/nbwipers

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.7.1 This release

11 files

0.7.0

11 files

0.6.2

11 files

0.6.1

11 files

0.6.0

11 files

0.5.1

11 files

0.5.0

11 files

0.4.0

11 files

0.3.7

11 files

0.3.6

12 files

0.3.5

12 files

0.3.4

12 files

0.3.3

12 files

0.3.2

12 files

0.3.1

12 files

0.3.0

12 files

0.2.0

12 files

0.1.1

12 files

0.1.0

12 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page