nbmodular 0.0.3

Convert notebooks to modular code

nbmodular

Convert data science notebooks with poor modularity to fully modular notebooks and / or python modules.

Motivation

In data science, it is usual to develop experimentally and quickly based on notebooks, with little regard to software engineering practices and modularity. It can become challenging to start working on someone else’s notebooks with no modularity in terms of separate functions, and a great degree of duplicated code between the different notebooks. This makes it difficult to understand the logic in terms of semantically separate units, see what are the commonalities and differences between the notebooks, and be able to extend, generalize, and configure the current solution.

Objectives

nbmodular is a library conceived with the objective of helping converting the cells of a notebook into separate functions with clear dependencies in terms of inputs and outputs. This is done though a combination of tools which semi-automatically understand the data-flow in the code, based on mild assumptions about its structure. It also helps test the current logic and compare it against a modularized solution, to make sure that the refactored code is equivalent to the original one.

Features

• Convert cells to functions.
• The logic of a single function can be written across multiple cells.
• Optional: processed cells can continue to operate as cells or be only used as functions from the moment they are converted.
• Create an additional pipeline function that provides the data-flow from the first to the last function call in the notebook.
• Write all the notebook functions to a separate python module.
• Compare the result of the pipeline with the result of running the original notebook.
• Converted functions act as nodes in a dependency graph. These nodes can optionally hold the values of local variables for inspection outside of the function. This is similar to having a single global scope, which is the original situation. Since this is memory-consuming, it is optional and may not be the default.
• Optional: Once we are able to construct a graph, we may be able to draw it or show it in text, and pass it to ADG processors that can run functions sequentially or in parallel.
• Persist the inputs and outputs of functions, so that we may decide to reuse previous results without running the whole notebook.
• Optional: if we have the dependency graph and persisted inputs / outputs, we may decide to only run those cells that are predecessors of the current one, i.e., the ones that provide the inputs needed by the current cell.
• Optional: if we associate a hash code to input data, we may only run the cells when the input data changes. Similarly, if we associate a hash code with AST-converted function code, we may only run those cells whose code has been updated.
• Optional: have a mechanism for indicating test examples that go into different test python files. = Optional: the output of a test cell can be used for assertions, where we require that the current output is the same as the original one.

Roadmap

• Convert cell code into functions:
  • Inputs are those variables detected in current cell and also detected in previous cells. This solution requires that created variables have unique names across the notebook. However, even if a new variable with the same name is defined inside the cell, the resulting function is still correct.
  • Outputs are, at this moment, all the variables detected in current cell that are also detected in posterior cells.
• Filter out outputs:
  • Variables detected in current cell, and also detected in previous cells, might not be needed as outputs of the current cell, if the current cell doesn’t modify those variables. To detect potential modifications:
    • AST:
      • If variable appears only on the right of assign statements or in if statements.
      • If it appears only as argument of functions which we know don’t modify the variable, such as print.
    • Comparing variable values before and after cell:
      • Good for small variables where doing a deep copy is not computationally expensive.
    • Using type checker:
      • Making the variable Final and using mypy or other type checker to see if it is modified in the code.
  • Provide hints:
    • Variables that come from other cells might not be needed as output. The remaining are most probably needed.
    • Variables that are modified are clearly needed.

Install

pip install nbmodular

Usage

Load ipython extension

Use ipython magic function by passing it the name of the function you want:

a = 2
b = 3
c = a+b
print (a+b)

5

This defines the function print_values, as follows:

</code></pre>
<pre><code>def print_values():
    a = 2
    b = 3
    c = a+b
    print (a+b)
    return a,b,c

Now, we can define another function in a cell that uses variables from the previous function.

d = 10

a = a + d
b = b + d
c = c + d
print (a, b, c, d)

12 13 15 10

</code></pre>
<pre><code>def add_new_values(idx, original_code, name, values_before, call, values_here, variables_here, variables_before, variables_after, arguments, return_values, code):
    a = a + d
    b = b + d
    c = c + d
    print (a, b, c, d)
    return d

By default, we can use variables from the previous cell as we normally do, i.e., values are still global. However, we can also opt to run the code encapsulated in