nbstripout-fast 1.0.0

Strips outputs from Jupyter notebooks

nbstripout-fast

A much faster version of nbstripout by writing it in rust (of course). This helps strip Jupyter Notebook output and metadata from notebooks. It is very useful as a git filter and is highly configurable.

Installation

pip install nbstripout-fast

Then replace nbstripout-fast with anywhere you use nbstripout.

Key differences

1. While we mirrored most of nbstripout's API, we do not support every nbstripout option.
2. There is no CLI option to install this in git for you
3. We support repository level settings in a .git-nbconfig.yaml file. Check out our examples. On a high level, you can add a git filter in a sitewide/user level and then allow each project to enforce consistent settings.

Why Rust?

nbstripout is a excellent project, but the python startup and import time makes its usage at scale a bit painful. While this means giving up on using nbconvert under the hood and ensuring the notebook is the correct format, it does make things up to 200x faster. This matters when you have a large number of files and git filter is called sometimes more than once per file. Let's look at the data:

Cells	nbstripout	nbstripout_fast
1	0m0.266s	0m0.003s
10	0m0.258s	0m0.003s
100	0m0.280s	0m0.004s
1000	0m0.372s	0m0.013s
10000	0m1.649s	0m0.133s

The table above shows a large overhead per notebook (mostly python startup time). When you have 100 or more notebooks, nbstripout takes more than 40s while nbstripout-fast takes only 1s!

Developing

You can use cargo which will build + run the CLI:

cargo run -- -t examples/example.ipynb

You can also build with cargo and run the script with the full path:

cargo build # dev build - ./target/debug/nbstripout-fast
cargo build --release # release build - ./target/release/nbstripout-fast

Running unit tests: maturin builds this repo to include pyo3 bindings by default. This allows for us to have an extension python extension mode as well. As of today, we can't have a binary and an extension, so we use the extension only for testing (issue).

pip install -e .
maturin develop
# Should output, this way you can use RUST_LOG=debug
in-venv pytest -rP

Debugging

Use RUST_LOG=debug to debug script for example:

RUST_LOG=debug cargo run -- '--extra-keys "metadata.bar cell.baz" -t foo.ipynb'

Releasing

We release only the bin version (i.e. just the CLI). The release builds wheels for manylinux. Before doing this, update the version in Cargo.toml

git tag vX.Y.Z
git push --tags

maturin build -b bin --sdist
twine upload -r target/wheels/*

History

This plugin was contributed back to the community by the D. E. Shaw group.

License

This project is released under a BSD-3-Clause license.

We love contributions! Before you can contribute, please sign and submit this Contributor License Agreement (CLA). This CLA is in place to protect all users of this project.