Parametrize and run scripts as notebooks with jupytext and papermill

This is on-going research on how to run scripts as notebooks using Jupytext and Papermill.

The corresponding GitHub issues are Jupytext #231 and Papermill #365.

Note that the below is a Bash Jupyter notebook that is tested on our CI. If you wish, you can also open and run it interactively on Binder.

cd demo

Jupytext and Papermill

It is possible to convert a script to a notebook using Jupytext, and then to run it using Papermill. Here we use --set-kernel - to use the kernel that matches the current Python environment.

jupytext script.py -o notebook.ipynb --set-kernel -

papermill notebook.ipynb executed_notebook.ipynb -p integer 2 -p text 'updated text'

python -c "import nbformat
with open('executed_notebook.ipynb') as fp:
    nb = nbformat.read(fp, as_version=4)
print(nb.cells[-1]['outputs'][0]['text'])"

Inject parameters in a script using Papermill

Papermill needs the language information in the kernel to inject the parameters in the notebooks, so in this case we also need to set a kernel for the notebook.

jupytext script.py -o notebook.ipynb --set-kernel -

papermill notebook.ipynb notebook_with_parameters.ipynb --prepare-only -p integer 3 -p text 'updated text, v3'

Once the parameters have been injected, we can convert back the notebook to a script, and drop the kernel information:

jupytext notebook_with_parameters.ipynb -o script_with_parameters.py --update-metadata '{"kernelspec":null, "jupytext":null}'

cat script_with_parameters.py

And finally, we run the script using the Python interpreter:

python script_with_parameters.py

Towards Papermill + Jupytext?

With the papermill_jupytext package, we can open Jupytext scripts with the txt:// address. A kernel pointing to the current Python environment is injected in the documents that have no kernel.

papermill txt://script.py executed_notebook.ipynb  -p integer 4 -p text 'updated text, v4'

python -c "import nbformat
with open('executed_notebook.ipynb') as fp:
    nb = nbformat.read(fp, as_version=4)
print(nb.cells[-1]['outputs'][0]['text'])"