📓 NotebookTester

A fast, reliable, and parallel Jupyter notebook testing tool with caching support! Test your notebooks with confidence.

🌟 Features

• 🚀 Parallel Execution: Test multiple notebooks simultaneously
• ⏱️ Configurable Timeouts: Set custom timeouts for notebook execution
• 💾 Smart Caching: Cache test results to avoid unnecessary re-runs
• 🎯 Selective Testing: Only test modified notebooks
• 📊 Detailed Logging: Comprehensive logs with test results and errors
• 🛠️ CLI Support: Easy to use command-line interface

🔧 Installation

Choose your preferred package manager:

Using pip (slow legacy)

pip install notebooktester

Using uv (10-100x faster)

see uv docs for more info.

uv add notebooktester

🚀 Quick Start

Test a single notebook:

notebooktester path/to/your/notebook.ipynb

Test all notebooks in a directory:

notebooktester path/to/notebooks/directory

Test a specific list of notebooks:

notebooktester nb1.ipynb nb2.ipynb nb3.ipynb

You can mix files and directories, and combine them with any of the options below:

notebooktester nb1.ipynb nb2.ipynb -t 120 -w 4

🎮 Command Line Options

notebooktester [OPTIONS] PATHS...

Options:

• -t, --timeout SECONDS: Timeout in seconds for each notebook (default: 60). With --report, the minimum duration to show (default: 30)
• -w, --workers NUMBER: Number of parallel workers (default: CPU count)
• -c, --cache-dir PATH: Cache directory for test results (default: .notebookcache)
• -v, --verbose: Enable verbose output (full log stream, no progress bar)
• -pb, --progressbar: Show a live, time-weighted progress bar based on cached notebook timings (mutually exclusive with -v)
• -f, --force: Ignore cache and force test execution
• -r, --report: Show long-running notebooks from the cache instead of running tests
• --version: Show the installed version and exit

Every run prints a summary line first, e.g. Testing 42 notebooks (30 cached/skipped, 12 to run) - estimated time ~3m10s, using timings from previous cached runs.

📋 Example Usage

Basic usage:

notebooktester notebooks/

With custom timeout and workers:

notebooktester notebooks/ -t 120 -w 4

Force re-run all tests:

notebooktester notebooks/ --force

Find long-running notebooks from the cache (no re-run needed):

notebooktester --report

Same, but with a lower threshold:

notebooktester --report -t 20

🔍 Cache Behavior

NotebookTester maintains a cache of test results to optimize performance:

• Only notebooks modified since their last test run are re-tested
• If the notebook timed out, and the current timeout has not been increased, notebook is skipped
• Cached results include success/failure status and error messages
• Force flag (-f) bypasses the cache

🐢 Long-Running Notebook Report

Once you've run the tests at least once, --report reads straight from the cache to show which notebooks are the slowest — no re-execution required. This pairs well with param() (see below): use the report to find your slow notebooks, then wrap their expensive parameters in param() so they run fast under notebooktester while keeping their real values everywhere else.

notebooktester --report
notebooktester --report -t 20        # lower the threshold to 20s
notebooktester --report -c .my-cache # point at a different cache dir

Example output:

>>notebooktester --report -t 20

🐢 Long-running notebooks (> 20s)

Notebook                        Time   Timeout  Status
------------------------------------------------------
notebooks/slow_train.ipynb     55.2s       60s  ✅ PASSED
notebooks/medium.ipynb         33.7s       60s  ✅ PASSED
------------------------------------------------------
2 notebook(s), 88.9s combined

⚡ Reducing Notebook Runtime with param()

Wrap expensive values in param(full, test=...) so notebooks run with a cheap stand-in under notebooktester, while keeping the real value everywhere else (e.g. when run interactively):

from notebooktester import param

EPOCHS = param(100, test=2)
SEARCH_SIZE = param(10_000, test=50)

📊 Output

The tool provides:

• Progress bar for test execution
• Colored console output for test results
• Detailed logs in the logs/ directory
• Summary of passed and failed tests

Example output:

>>notebooktester notebooks/ -t 120 -w 4
Running tests with 4 workers
Starting notebook tests - found 10 notebooks
✅ PASSED - notebook1.ipynb: Success
❌ FAILED - notebook2.ipynb: Cell execution error
⏰ TIMEOUT - notebook3.ipynb: A cell timed out

Test Summary: 8 passed, 2 failed

🤝 Contributing

Contributions are welcome! Feel free to:

• Report bugs
• Suggest features
• Submit pull requests

👤 Author

Raoul Grouls