ipyflow 0.0.157

Next-generation IPython kernel with reactivity, execution suggestions, syntax extensions, and more.

IPyflow

Install

pip install ipyflow

About

ipyflow is a next-generation Python kernel for Jupyter and other notebook interfaces that tracks dataflow relationships between symbols and cells during a given interactive session. Here are some of its main features:

Execution Suggestions

To keep the execution state consistent with the code in cells, rerun the turquoise-highlighted cells, and avoid the red-highlighted cells:

A turquoise-highlighted input with red-highlighted output just means that the output may be out-of-sync.

Reactivity

Do you trust me? Good. It's time to free yourself of the burden of manual re-execution. Use ctrl+shift+enter (on Mac, cmd+shift+enter also works) to execute a cell and its (recursive) dependencies reactively:

You can also run the magic command %flow mode reactive in any cell to enable reactivity as the default execution mode:

Disable by running %flow mode normal.

Syntax Extensions

Prefixing a symbol with $ in a load context will cause the referencing cell to re-execute itself, whenever the aforementioned symbol is updated (regardless of execution mode):

You can also use the $ syntax in store contexts, which triggers cells that reference the corresponding symbol to re-execute, regardless of whether the reference is similarly $-prefixed:

Finally, you can also prefix with $$ to trigger a cascading reactive update to all dependencies in the chain, recursively:

Integration with ipywidgets

ipyflow's reactive execution engine, as well as its APIs (see "State API" below) are fully compatible with ipywidgets, allowing cells to respond to slider changes, button clicks, and other events:

This functionality can be paired with other extensions like stickyland to build fully reactive dashboards on top of JupyterLab + ipyflow.

State API

ipyflow must understand the underlying execution state at a deep level in order to provide its features. It exposes an API for interacting with some of this state, including a code function for obtaining the code necessary to reconstruct some symbol:

# Cell 1
from ipyflow import code

# Cell 2
x = 0

# Cell 3
y = x + 1

# Cell 4
print(code(y))

# Output:
"""
# Cell 2
x = 0

# Cell 3
y = x + 1
"""

You can also see the cell (1-indexed) and statement (0-indexed) of when a symbol was last updated with the timestamp function:

from ipyflow import timestamp
timestamp(y)
# Timestamp(cell_num=3, stmt_num=0)

To see dependencies and dependents of a particular symbol, use the deps and users fuctions, respectively:

from ipyflow import deps, users

deps(y)
# [<x>]

users(x)
# [<y>]

If you want to elevate a symbol to the representation used internally by ipyflow, use the lift function (at your own risk, of course):

from ipyflow import lift

y_sym = lift(y)
y_sym.timestamp
# Timestamp(cell_num=3, stmt_num=0)

Finally, ipyflow also comes with some rudimentary support for watchpoints:

# Cell 1
from ipyflow import watchpoints

def watchpoint(obj, position, symbol_name):
    if obj <= 42:
        return
    cell, line = position
    print(f"{symbol_name} = {obj} exceeds 42 at {cell=}, {line=}")

# Cell 2
y = 14
watchpoints(y).add(watchpoint)

# Cell 3
y += 10

# Cell 4
y += 20
# y = 44 exceeds 42 at cell=4, line=1

Quick Start

To install, run:

pip install ipyflow

To run an ipyflow kernel in JupyterLab, select "Python 3 (ipyflow)" from the list of available kernels in the Launcher tab. For classic Jupyter, similarly select "Python 3 (ipyflow)" from the list of notebook types in the "New" dropdown dialogue.

JupyterLab Entrypoint:	Classic Jupyter Entrypoint:

Note: reactive execution features are not yet supported in classic Jupyter notebooks.

Citing

ipyflow started its life under the name nbsafety, which provided the initial suggestions and slicing functionality.

For the execution suggestions:

@article{macke2021fine,
  title={Fine-grained lineage for safer notebook interactions},
  author={Macke, Stephen and Gong, Hongpu and Lee, Doris Jung-Lin and Head, Andrew and Xin, Doris and Parameswaran, Aditya},
  journal={Proceedings of the VLDB Endowment},
  volume={14},
  number={6},
  pages={1093--1101},
  year={2021},
  publisher={VLDB Endowment}
}

For the dynamic slicer (used for reactivity and for the code function, for example):

@article{shankar2022bolt,
  title={Bolt-on, Compact, and Rapid Program Slicing for Notebooks},
  author={Shankar, Shreya and Macke, Stephen and Chasins, Andrew and Head, Andrew and Parameswaran, Aditya},
  journal={Proceedings of the VLDB Endowment},
  volume={15},
  number={13},
  pages={4038--4047},
  year={2022},
  publisher={VLDB Endowment}
}

We don't have a paper written yet for the syntax extensions that implement the reactive algebra, but in the mean time, you can cite the ipyflow repo directly for that and anything else not covered by the previous publications:

@misc{ipyflow,
  title = {{IPyflow: A Next-Generation, Dataflow-Aware IPython Kernel}},
  howpublished = {\url{https://github.com/ipyflow/ipyflow}},
  year = {2022},
}

License

Code in this project licensed under the BSD-3-Clause License.