Skip to main content

Microservice to generate Jupyter reports

Project description

Papermill Jupyter report

Install with conda Github Actions Status

Microservice to generate Jupyter reports by executing notebook and exporting them to static HTML page combining papermill and nbconvert.

It can be run as a standalone application or as a JupyterHub service.

Demo

The workflow is described in the figure below. You can either:

  1. Set a specific URL to select a notebook and pass query arguments as parameters,
  2. Or interactively select one notebook and set its parameters.

Features

The API is described there.

There are two features provided by this service:

  • Listing all available report templates (and their parameters) The available templates are all notebook files existing within the template_root_dir.
  • Generate a report (i.e. execute a parametrized notebook and convert it to HTML)

Parametrized notebook are supported only for Python notebook.

Configuration

The configurable settings for the service are:

  • broken_reports_dir: Folder in which broken notebook will be copied - it must be a subfolder of notebook_dir; default /home/USERNAME/broken_reports
  • config_file: Configuration file name; default papermill_service_config
  • git_auth: Git authentication (username:password); default None
  • notebook_dir: Notebook server root directory; default /home/USERNAME This is needed to build the link to broken notebook.
  • port: Port of the service; default 8888
  • template_root_dir: Folder containing the notebook templates on the server; default /opt/papermill_report
  • template_dir: Folder of the Git repository containing the notebook templates; default "."
  • template_git_url: Git repository URL source of the notebook templates; default None
  • template_paths: Paths to search for service webpage jinja templates, before using the default templates; default None

The string USERNAME will be replaced with the user’s username if used in broken_reports_dir or notebook_dir.

Notes

  • The service must be run as root because report processes are executed through su <user> --login command to impersonate the authenticated user and setting the environment variables afresh.

Development

This python package is meant to be deployed as a JupyterHub Hub-Managed service.

The consequences are:

  • The service runs its own tornado server. Requests will be forwarded to it by the JupyterHub internal proxy from the standard URL https://myhub.horse/services/my-service/ (pay attention to the required trailing /).
  • Authentication is deferred to JupyterHub
  • As it is managed by JupyterHub, JupyterHub will check that the service is alive. And if not, it will restart it. Moreover, when JupyterHub is nicely stopped, it will stop the service.

Testing

There is 2 levels of test in this service. Some conventional unit tests with pytest and a Dockerfile to spin an integrated environment starting the service as JupyterHub service.

Unit test

python -m install -r requirements.txt -r requirements_dev.txt
pytest

Integration with JupyterHub

To build and launch the integrated environment:

docker build -t papermill-report .
docker run -p 8000:8000 --rm papermill-report

The Hub is parameterized (see jupyterhub_config.py) with two users:

  • jovyan: an administrator
  • marc: an user

There are no password on the accounts.

The template folder is the examples folder of this project.

To test the service, go to one valid endpoint:

  • http://localhost:8000/services/report/
  • http://localhost:8000/services/report/broken_parameters.ipynb
  • http://localhost:8000/services/report/no_parameters.ipynb
  • http://localhost:8000/services/report/subfolder/simple_execute.ipynb&msg=hello

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

papermill_report-2.0.3.tar.gz (702.4 kB view hashes)

Uploaded Source

Built Distribution

papermill_report-2.0.3-py2.py3-none-any.whl (60.4 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page