Skip to main content

A toolset to accelerate Jupyter notebook workflows

Project description

Conveyor

Build Status

Compute and use Jupyter notebook cell outputs in other notebooks and scripts with just a couple lines of code.

Use Cases

  • Split ordered steps across multiple notebooks, picking up the most recent state information with each new notebook
  • Prototype and organize workflows entirely in Jupyter notebooks without having to manage script exports
  • Improve performance by reducing overheads from notebook servers and unused code cells

Requirements

Conveyor currently only supports Jupyter notebooks written in Python. It is important to ensure that you have the version of Python (and any dependencies) used in your notebooks also installed locally, or on the machine using this library.

Examples

For quick and simple access to values in a prior notebook:

conveyor.run_notebook("conveyor/examples/Sample Calculations I.ipynb", import_globals=True)

# all notebook globals pushed to conveyor.nbglobals with import_globals flag
from conveyor.nbglobals import x, y, z, fig

print(y)

Conveyor can also provide you with any information available in a prior Jupyter workspace.

results = conveyor.run_notebook("conveyor/examples/Sample Calculations I.ipynb")

# Cells are zero-indexed, only code cells are counted
code_cell_idx = 1

# Get a cell's source code
code = results[code_cell_idx]['code']

# Get a cell result
cell_result = results[code_cell_idx]['result']

# Get cell stdout
cell_stdout = results[code_cell_idx]['stdout']

# To get any variable available in notebook
x = results.getvar('x')

There are options for running notebooks that can be used to optimize notebook execution, or stitch notebooks together in pipelines.

from conveyor.multinb import Pipeline
...
data_processing = Pipeline()
data_processing.add_notebook(filename="conveyor/examples/load_data.ipynb", carry_vars=['df'])
data_processing.add_notebook(filename="conveyor/examples/process_data.ipynb", 
    carry_vars=['magic_number'], start_cell_idx=3)

# Add custom intermediary steps
def transform_magic(from_state):
    to_state = dict()
    to_state['transformed_magic_number'] = -1 * from_state['magic_number']
    return to_state

data_processing.add_transform(transform_magic)

# Get output of selected variables from each stage
results = data_processing.run()

Installation

This package is available on pypi. Install it using pip with

pip install jupyter-conveyor

Conveyor is only compatible with Python 3.5 and above.

Documentation

See the docs folder, or read them here.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for jupyter-conveyor, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size jupyter-conveyor-0.0.3.tar.gz (7.7 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page