nb-workflows 0.6.0

Schedule parameterized notebooks programmatically using cli or a REST API

NB Workflows

Description

If SQL is a lingua franca for querying data, Jupyter should be a lingua franca for data explorations, model training, and complex and unique tasks related to data.

NB Workflows is a library and a platform that allows you to run parameterized notebooks in a distributed way. A Notebook could be launched remotly on demand, or could be schedule by intervals or using cron syntax.

Internally it uses Sanic as web server, papermill as notebook executor, an RQ for task distributions and coordination.

Goal

Empowering different data roles in a project to put code into production, simplifying the time required to do so. It enables people to go from a data exploration instance to an entirely pipeline deployed in production, using the same notebook file made by a data scientist, analyst or whatever role working with data in an iterative way.

Features

• Define a notebook like a function, and execute it on demand
• Automatic Dockerfile generation. A project should share a unique environment
• Docker building and versioning: it build and track each release.
• Execution History, Notifications to Slack or Discord.

Roadmap

See Roadmap draft

Architecture

References & inspirations

• Notebook Innovation - Netflix
• Tensorflow metastore
• Maintainable and collaborative pipelines