Strips outputs from Jupyter notebooks

Navigation

Verified details

These details have been verified by PyPI
Project links
Owner
GitHub Statistics
Maintainers
Avatar for divyansshhh from gravatar.com divyansshhh

Unverified details

These details have not been verified by PyPI
Classifiers
Report project as malware

Project description

nbstripout-fast

PyPI version PyPI DM Github Actions Status

A much faster version of nbstripout by writing it in rust (of course). This helps strip Jupyter Notebook output and metadata from notebooks. It is very useful as a git filter and is highly configurable.

Installation

pip install nbstripout-fast

Then replace nbstripout-fast with anywhere you use nbstripout.

Key differences

  1. While we mirrored most of nbstripout's API, we do not support every nbstripout option.
  2. There is no CLI option to install this in git for you
  3. We support repository level settings in a .git-nbconfig.yaml file. Check out our examples. On a high level, you can add a git filter in a sitewide/user level and then allow each project to enforce consistent settings.

Why Rust?

nbstripout is a excellent project, but the python startup and import time makes its usage at scale a bit painful. While this means giving up on using nbconvert under the hood and ensuring the notebook is the correct format, it does make things up to 200x faster. This matters when you have a large number of files and git filter is called sometimes more than once per file. Let's look at the data:

Cells nbstripout nbstripout_fast
1 0m0.266s 0m0.003s
10 0m0.258s 0m0.003s
100 0m0.280s 0m0.004s
1000 0m0.372s 0m0.013s
10000 0m1.649s 0m0.133s

The table above shows a large overhead per notebook (mostly python startup time). When you have 100 or more notebooks, nbstripout takes more than 40s while nbstripout-fast takes only 1s!

Example

This example illustrates how nbstripout-fast can be used to automatically clean Jupyter notebooks using Git filters (see e.g. Git Attributes). This keeps your repository clean by removing unnecessary output and clutter, while preserving your local working version. The benefits are minimised diffs and reduced repository size.

  1. Install nbstripout-fast as described above.

  2. Configure nbstripout-fast

    Create a .git-nbconfig.yaml file at the root of your repository to configure nbstripout-fast, e.g.

    nbstripout_fast:
  keep_count: false
  keep_output: false
  drop_empty_cells: true
  extra_keys: []
  keep_keys: []

  3. Set Git Attributes

    Create a .gitattributes file at the root of your repository if it doesn't yet exist and add this line:

    *.ipynb filter=jupyter

    This instructs Git to use a custom filter named "jupyter" on all .ipynb files.

  4. Configure the jupyter Filter

    Run these commands in your terminal to configure the "jupyter" filter:

    git config filter.jupyter.clean nbstripout-fast
git config filter.jupyter.smudge cat
  • clean: This filter runs nbstripout-fast when adding notebooks to the version that is checked out, i.e. the clean version.
  • smudge: This filter runs cat when checking out notebooks, ensuring your local (smudged) version remains unmodified. Git filters transform files at the time of checkout and commit.

  1. Reapply Cleaning to Existing Notebooks (Optional)

    If you already have Jupyter notebooks tracked by Git, you can reapply the cleaning process to them:

    git add --renormalize . git commit -m "Cleaned Jupyter notebooks"

Stripping specific cell outputs

To strip cell outputs that match a regular expression, the --strip-regex option can be used in combination with --keep-output. For example, to remove cell outputs that only contain a notebook widget:

nbstripout-fast --keep-output --strip-regex "^Output\(\)$"

or to remove completed tqdm progress bars:

nbstripout-fast --keep-output --strip-regex "100%.*"

See the documentation for regex for information about supported regex syntax.

Developing

You can use cargo which will build + run the CLI:

cargo run -- -t examples/example.ipynb

You can also build with cargo and run the script with the full path:

cargo build # dev build - ./target/debug/nbstripout-fast
cargo build --release # release build - ./target/release/nbstripout-fast

Running unit tests: maturin builds this repo to include pyo3 bindings by default. This allows for us to have an extension python extension mode as well. As of today, we can't have a binary and an extension, so we use the extension only for testing (issue).

pip install -e .
maturin develop
# Should output, this way you can use RUST_LOG=debug
in-venv pytest -rP

Debugging

Use RUST_LOG=debug to debug script for example:

RUST_LOG=debug cargo run -- '--extra-keys "metadata.bar cell.baz" -t foo.ipynb'

Releasing

Manylinux, macos, and windows wheels and sdist are built by github workflows. Builds are triggered upon the creation of a pull request, creating a new release, or with a manual workflow dispatch. The wheels and sdist are only uploaded to PyPI when a new release is published. In order to create a new release:

  1. Create a commit updating the version in Cargo.toml and CHANGELOG.md, then create a git tag:
git tag vX.Y.Z
git push --tags
  1. Draft a new release in github; select the tag that you just created.
  2. Once the new release is created, the wheels and sdist will be built by a github workflow and then uploaded to PyPI automatically using the PYPI_API_TOKEN in the github secrets for the repository.

History

This plugin was contributed back to the community by the D. E. Shaw group.

D. E. Shaw Logo

License

This project is released under a BSD-3-Clause license.

We love contributions! Before you can contribute, please sign and submit this Contributor License Agreement (CLA). This CLA is in place to protect all users of this project.

Project details

Verified details

These details have been verified by PyPI
Project links
Owner
GitHub Statistics
Maintainers
Avatar for divyansshhh from gravatar.com divyansshhh

Unverified details

These details have not been verified by PyPI
Classifiers

Release history Release notifications | RSS feed

This version

1.1.1

1.1.0

1.0.4

1.0.3

1.0.2

1.0.1

1.0.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nbstripout_fast-1.1.1.tar.gz (28.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

nbstripout_fast-1.1.1-py3-none-win_amd64.whl (1.1 MB view details)

Uploaded Python 3Windows x86-64

nbstripout_fast-1.1.1-py3-none-win32.whl (1.0 MB view details)

Uploaded Python 3Windows x86

nbstripout_fast-1.1.1-py3-none-musllinux_1_2_x86_64.whl (1.5 MB view details)

Uploaded Python 3musllinux: musl 1.2+ x86-64

nbstripout_fast-1.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

nbstripout_fast-1.1.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl (1.4 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ i686

nbstripout_fast-1.1.1-py3-none-macosx_11_0_arm64.whl (1.2 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

File details

Details for the file nbstripout_fast-1.1.1.tar.gz.

File metadata

  • Download URL: nbstripout_fast-1.1.1.tar.gz
  • Upload date:
  • Size: 28.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.8

File hashes

Hashes for nbstripout_fast-1.1.1.tar.gz
Algorithm Hash digest
SHA256 54cb455ab1326fa99d446f4c92f8359fa5f31351b02a6e286caf81334fa8266c
MD5 d9d2cf227942330e8feca0d2aef24748
BLAKE2b-256 d81454aa0d3849747ce59650455f328dc471f3fefa0aa901cf3f6a859e54faf8

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbstripout_fast-1.1.1.tar.gz:

Publisher: wheels.yml on deshaw/nbstripout-fast

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbstripout_fast-1.1.1-py3-none-win_amd64.whl.

File metadata

File hashes

Hashes for nbstripout_fast-1.1.1-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 d2b79eeef8a3859db76f009bd2be066c0046d8b68b48e908d5e295b4f2c1d2c2
MD5 4c6fdc2b499a41031245673a6a20f61f
BLAKE2b-256 3daa26889da6e5381cbe17709f3392e153585c1fcf5af3f96d7cf81ca9acafc0

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbstripout_fast-1.1.1-py3-none-win_amd64.whl:

Publisher: wheels.yml on deshaw/nbstripout-fast

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbstripout_fast-1.1.1-py3-none-win32.whl.

File metadata

  • Download URL: nbstripout_fast-1.1.1-py3-none-win32.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: Python 3, Windows x86
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.8

File hashes

Hashes for nbstripout_fast-1.1.1-py3-none-win32.whl
Algorithm Hash digest
SHA256 d960cf9c1daaf2d981bbc7457076ec05d3cddfa122297bc09deab628b9b70b75
MD5 37323958a587ed4484e40999b3ee6a14
BLAKE2b-256 835ee5d7ddfae1779723815aea0fb45536679369268686839471b39026aaadab

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbstripout_fast-1.1.1-py3-none-win32.whl:

Publisher: wheels.yml on deshaw/nbstripout-fast

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbstripout_fast-1.1.1-py3-none-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for nbstripout_fast-1.1.1-py3-none-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 23cfa4450e93ec0dd553b37f6cce3c51f7877b636865fd44a3c828f8589e68f9
MD5 895bccf50b57e38501fbd6ae56e25b19
BLAKE2b-256 0416511ea327bc216750abb5a3a8b620ba79873e92593f55ac3acffbfe3cea69

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbstripout_fast-1.1.1-py3-none-musllinux_1_2_x86_64.whl:

Publisher: wheels.yml on deshaw/nbstripout-fast

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbstripout_fast-1.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for nbstripout_fast-1.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 5b89f7c5b36a8e95d54a72021ec327d6517424bc5ac588f17d327966a9b46701
MD5 b038a37aac86b313ead5744eb30d06e8
BLAKE2b-256 2b0bed28a5942b0616331f0fa3edf8452b411e0e78f6c947d9bc80753dd12c98

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbstripout_fast-1.1.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: wheels.yml on deshaw/nbstripout-fast

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbstripout_fast-1.1.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for nbstripout_fast-1.1.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 eba123f9b1799a49666fad9c0efd46bef4bc7f709e47a8ca0685ad335f903fc0
MD5 1fb2eedf993e37d012e5f8b509334daf
BLAKE2b-256 e5e0a460719c0ed8a18c897300fea0154db00e510000a118b5d898cffbe2028e

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbstripout_fast-1.1.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl:

Publisher: wheels.yml on deshaw/nbstripout-fast

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nbstripout_fast-1.1.1-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for nbstripout_fast-1.1.1-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 8c8cd2a2446e1334b107ba9b65e8e4e0f74380b1884e5e96bf0d06e0e5cdb21d
MD5 53dd59a6ba78361b5bbcde87228fff2d
BLAKE2b-256 7eed5efa29427668e665ca00b98d521669e30f08f1b1f23dd9c777b604532342

See more details on using hashes here.

Provenance

The following attestation bundles were made for nbstripout_fast-1.1.1-py3-none-macosx_11_0_arm64.whl:

Publisher: wheels.yml on deshaw/nbstripout-fast

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page