jnbv 2021.10.26

Validation of Jupyter notebooks and kernels

JUPYTER NOTEBOOK VALIDATION

1. Project Overview
2. Install
3. Usage
4. Contributing

PROJECT OVERVIEW

This repository contains scripts used to validate ipynb notebooks and Jupyter kernels.

There is another related repository which contains a large number of test ipynb notebooks and routines to help with the execution of the validation tests via docker images and slurm batch jobs:

• jupyter-notebook-validation

PACKAGE

A python package has been created which can be installed via pip or conda:

• conda package:
• PyPi package:

GOALS

The goals of this project are to be able to:

1. Execute validation of Jupyter kernels using ipynb notebooks in the terminal
2. Execute validations non-interactively in a CI pipeline
3. Check execution output for errors
4. Compare output of an execution to known output
5. Select different Jupyter kernels to use
6. Log results of notebook executions and tests

EXAMPLE OUTPUT

Partial output from validation tests run in the terminal:

Output from an ipynb notebook created from failed execution of a notebook:

INSTALL

The jnbv module can be installed with pip or conda using one of the following:

• conda package:
• PyPi package:
• gitlab repository:

INSTALL WITH PIP

Create a virtual environment if you don't already have one:

python3 -m venv venv

Activate your environment, in this example it's in venv/:

source venv/bin/activate

Install the code from PyPi using pip:

pip install jnbv

Or install from gitlab using pip and git:

pip install git+https://gitlab.com/MAXIV-SCISW/JUPYTERHUB/jnbv.git

With either method, dependencies will also be installed.

INSTALL WITH CONDA

Activate and install directly into your environment:

source venv/bin/activate
conda install jnbv -c conda-forge

Or, you can add jnbv via your conda environment yaml file, by adding conda-forge to your list of channels, and jnbv to your list of dependencies, as in this example file:

---
name: my-conda-env
channels:
  - conda-forge
dependencies:
  - jnbv

Then update your conda environment, for example the base conda environment which is in the directory venv/:

venv/bin/conda env update --name base --file my-conda-env.yml

USAGE

jnbv -h

EXECUTE NOTEBOOK

For this, you will need to have:

1. A base environment in which jnbv has been installed (see above)
2. An ipynb notebook file
3. A Jupyter kernel that will be used to execute the file

Once you have all these items, the base environment should be activated:

source venv/bin/activate

Check to see what kernels you have available:

jupyter kernelspec list
    Available kernels:
      hdf5-kernel    /var/www/jupyterhub/jnbv/venv/share/jupyter/kernels/hdf5-kernel
      python3        /var/www/jupyterhub/jnbv/venv/share/jupyter/kernels/python3

If you don't have any kernels, then install ipykernel into your environment with either pip or conda:

pip install ipykernel
conda install ipykernel

and then you should have the default kernel "python3" available.

If you don't have an ipynb notebook handy, you can get an example notebook file here:

wget https://gitlab.com/MAXIV-SCISW/JUPYTERHUB/jnbv/-/raw/master/development/the-meaning-of-life.ipynb

And then the ipynb notebook can be executed in the terminal, using the default kernel python3 for example:

jnbv the-meaning-of-life.ipynb \
    --kernel_name python3 \
    --execute

READ NOTEBOOK

By defualt, the result of executing an ipynb file is a new ipynb file named output.ipynb. It can be read in the terminal with:

jnbv output.ipynb --read

TEST NOTEBOOK

The same file can be checked for errors:

jnbv output.ipynb --test

COMPARE NOTEBOOKS

And two ipynb notebooks can be compared:

jnbv output.ipynb --compare the-meaning-of-life.ipynb

EXECUTE, TEST, COMPARE, SAVE

All steps can be made to run in succession with one command:

jnbv the-meaning-of-life.ipynb \
    --kernel_name python3 \
    --execute --read --test --compare --save

Or, more simply using the --validate option, which is a combination of 5 other options:

jnbv the-meaning-of-life.ipynb \
    --kernel_name python3 \
    --validate

Note that above the option --save was also added, which then creates the output directory test-results/, and within that creates subdirectories with kernel names, date stamps, and finally log files and new ipynb files.
For example:

test-results/
└── python3/
    └── 2021-06-11_14-39-40/
        ├── the-meaning-of-life.ipynb
        └── the-meaning-of-life.log

Example output from executing a simple notebook using the default kernel:

CONTRIBUTING

Would you like to contribute to this project? See: CONTRIBUTING.md

JNBV CHANGELOG

2021.10.26 (2021-10-26)

• added to list of source to ignore View
• updateed execution of test notebook View
• updated the execution of a couple notebooks View

2021.8.26 (2021-08-26)

• updated license View
• trying to setup sast View
• try adding script to SAST job View
• trying again View
• added artifacts View
• try adding a gl-sast-report.json file View
• try adding gl-sast-report.json View
• trying some other things... View
• trying something else View
• trying appthreat/sast-scan View
• changed before_script for sast job View
• changed type to python View
• changed job name View
• trying semgrep View
• added berfore_script to semgrep job View
• change to semgrap command View
• try some other semgrep settings View
• other semgrep settings View
• trying to view artifact View
• trying other semgrep settings View
• trying bandit for security testing, replaced assert statements with raise AssertionError View
• added pip install View
• adding more to bandit checks View
• changed asserrt statment to comply with bandit errors View
• trying different bandit command View
• change to bandit scan trigger View
• added creation of bandit reports View
• bumped version in setup.cfg to 2021.8.26 View

2021.8.18 (2021-08-18)

• updated readme View
• added a source line to ignore View
• added combining of stderr outputs View
• bumped version in setup.cfg to 2021.8.18 View

2021.8.17.1 (2021-08-17)

• added retry of ntoebook execution in case of kernel dying View
• added comparison test, fixed mistake in execution function View
• added special validation notebook for py36 View
• bumped version in setup.cfg to 2021.8.17.1 View

2021.8.17 (2021-08-17)

• updated screenshot View
• added warning to ignore View
• bumped version in setup.cfg to 2021.8.17 View
• bumped version in setup.cfg to 2021.8.17.1 View
• slight change to versioning setup View
• fixed typo View
• bumped version in setup.cfg to 2021.8.17 View

2021.8.16.0 (2021-08-16)

• small changes to comments View
• added function for removal of warnings in cell output View
• small change to output View
• removed old comments View
• added a couple tests and dependencies needed in kernel environments View
• had forgotten to add removal of warnings from both outputs when doing comparisons View
• added another wanring to ignore, added check for existance of subkey View
• bumped version in setup.cfg to 2021.8.16.0 View

2021.8.4.0 (2021-08-04)

• reorganization of tests in tox configuration View
• small edits of documentation View
• bumped version in setup.cfg to 2021.8.4.0 View

2021.8.3.0 (2021-08-03)

• added makefile targets for new tests, added badges to top of readme View
• updated documentation a bit View
• edited screenshots slightly View
• added tests for reading of notebooks View
• try to make reading tests fail View
• change to exection CI test triggers View
• added tests of test_ipynb function View
• fixed incorrect asserion tests View
• added a bit to the documentation View
• added simple changelog generator target to makefile View
• added comparison tests, combined some test executions in tox config View
• fixed typo, changed some CI job names View
• combined some tests in tox config View
• changed test function names View
• changed some test names View
• fixed incorrect test name in CI config View
• slight comment change View
• added some comments View
• small change to comparison code, added a test View
• simplified validation tests, made comparison check for newline better View
• added some more execution and validation tests View
• trying to get varsioning, tagging, and changelog to all work nice together View
• bumped version in setup.cfg to 2021.8.3.0 View
• bumped version in setup.cfg to 2021.8.3.0 View

2021.7.28.0 (2021-07-28)

• cropped screenshots View
• created pytest for execution View
• fixed mistake in tox test, added to documentation View
• fixed some documentation mistakes View
• small edits of documentation View
• change to CI triggers, slimmed test_execute_success_hdf5_kernel() View
• added various tests of notebook execution View
• some changes to execution tests View
• fixed incorrect names View
• changed test names, making CI more compact View
• bumped version in setup.cfg to 2021.7.28.0 View

2021.7.21.3 (2021-07-21)

• updated documentation View
• updated documentation View
• bumped version in setup.cfg to 2021.7.21.3 View

2021.7.21.2 (2021-07-21)

• created makefile target to upload to PyPi, altered CI to upload to PyPi View
• bumped version in setup.cfg to 2021.7.21.2 View

2021.7.21.1 (2021-07-21)

• bumped version in setup.cfg to 2021.7.21.1 View

2021.7.21.0 (2021-07-21)

• have commit counts start at 0 instead of 1 View
• bumped version in setup.cfg to 2021.7.21.0 View

2021.7.19.9 (2021-07-19)

• updated documentation View
• bumped version in setup.cfg to 2021.7.19.9 View

2021.7.19.8 (2021-07-19)

• bumped version in setup.cfg to 2021.7.19.8 View

2021.7.19.7 (2021-07-19)

• made link to CONTRIBUTING.md accessible from pypi page View
• bumped version in setup.cfg to 2021.7.19 View
• bumped version in setup.cfg to 2021.7.19_8fe57776 View
• bumped version in setup.cfg to 2021.7.19.7976e902 View
• bumped version in setup.cfg to 2021.7.19.dev6ccdf2a1 View
• bumped version in setup.cfg to 2021.7.19_5 View
• bumped version in setup.cfg to 2021.7.19-6 View
• bumped version in setup.cfg to 2021.7.19.7 View

2021.7.19 (2021-07-19)

• updated readme, added comments to yaml files View
• fixed typos View
• changed triggers for some CI stages View
• making png image pypi friendly View
• bumped version in setup.cfg to 2021.7.19 View

2021.7.16 (2021-07-16)

• re-grouping of makefile targets, new screenshot View
• updated documentation View
• small change to target and stage name View
• added a bit to the documentation View
• bumped version in setup.cfg to 2021.7.16 View

2021.7.15 (2021-07-15)

• small comment edit View
• created test of jnbv terminal execution View
• added new jnbv temrinal execution test to CI View
• try unpinning version in env yaml View
• remove basepython configuration View
• removed some modules, try to figure out issue View
• try adding back in validation test View
• ipykenrel is needed, but seems to be the source of issue when installing in gitlab CI View
• try adding validation test again View
• cleaned up some comments, changed makefile target names View
• created a validation test for use with pytest View
• pytest excution working with all versions of python View
• fixed incorrect test env name View
• split testing environments - one for basic building, the other for testing of a kernel View
• updated instructions and screenshot View
• bumped version in setup.cfg to 2021.7.15 View

2021.7.12 (2021-07-12)

• some cleaning up of nested if-statements, added example hdf5 kernel, notebook, data View
• some more cleaning up View
• max-complexity in flake8 now 10 View
• a bit more if-nesting removal View
• added sourcing of conda env prior to test execution View
• bumped version in setup.cfg to 2021.7.12 View

BSD 2-Clause License

Copyright (c) 2021, Jason Brudvik, MAX IV Laboratory

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.