QiTangle - IPython magic to entangle code from notebooks.

export code from notebook cells to python source files

The first Quarto document I created was used to implement a python module based on some math formula’s from an article (about measuring sap flux in tree stems). Quarto let me follow the article and methodically, step by step functions were build and documented along the formula’s and some example calculations. This worked like a charm. When everything was implemented and working all there was left to do was copy the code from the document into a separate python file for use in our runtime environment.

Any changes in eihter of the two sources - the notebook and the script - had to be copied from one to theother. Not an ideal process, what I needed was a way to convert or export selected code cells into a python source code file automatically. Initially I found this in NBDev. NBDev exports your code, builds the documentation and, if desired, packages the code and uploads to PyPi.

Jupyter notebooks, the primary document format of NBDev, are uncomfortable to use in combination with git, even with the clean-up tools provided by NBDev, many unneeded git diffs were checked in and worse; merge conflicts can be a huge problem. Working accross multiple documents with repeated export, renders and restarts to refresh imports of updated exports was less then ideal.

Something closer to using Quarto’s own markdown was desired, simply extracting or exporting python sourcefiles from Quarto documents seemed sufficient. Quarto’s interactive python support is based on IPython and Jupyter kernels, and these offer an extension mechanism called magics. These extensions are available in Jupyter and Quarto notebooks and offered oppurtunities.

So I implemented IPython magic to export code from notebook cells to python source files and called it QiTangle after Quarto, IPython and TANGLE, the latter being the name of a component and process in Knuth’s Literate programming system that was responsible for tangling the computer code together.

QiTangle and literate programming

Quarto’s rich document creation and interactive computing extended with the ability to export selected Python code cells into sourcecode file are not entirely dissimilar to Knuth’s Literate programming paradigm. WEB’s system allowed for describring pieces of computer code and algorithms that could be combined with each other. QiTangle is more restricted in that it only allows chunks of lines of code to be exported to a python source file, without combining expressions. It does however, unlike many similar notebook based systems, allow cells to be reordered into the exported source file. As in Knuth’s system, QiTangle allows disconnecting the narrative order that descibes the code in an order suitable to explain it, from the neccesities imposed on the ordering of code by the computer language.

Qi is a vital force traditionally believed to be a part of all living entities. Literally meaning ‘vapor’, ‘air’, ‘gas’, or ‘breath’, the word qi is polysemous, often translated as ‘vital energy’, ‘vital force’, ‘material energy’, or simply ‘energy’. (from Wikipedia)

Installing

pip install qitangle

Usage

Loading the extension

import qitangle
%reload_ext qitangle.entangle

Using %reload_ext will reset the cell collection even when a kernel process remains active as is the case with Quarto

Tangling

Tangling is the process of exporting the individual code cells to a file in the order as specified by the label and dependencies attributes of an entangled cell.

Each %%entangle invocation adds or updates a code-cell in a collection, it’s label and dependencies inform the %tangle which cells depend on which other cells and thereby ultimately the order in which they are exported. By exporting it’s dependencies before the cell we ensure that prior to excution of that cell everything, well it’s code dependencies at least, are in place.

When updating results in a collection that can not be ordered correctly cell execution is delayed until their dependencies are added.

Optionally each cell individually can be reformatted through autopep8 and optionally when exporting, the entire produced file can be reformatted. Configuration of autopep8 is described in it’s manual.

Invocations

Line level magic

Line magic %tangle controls the tangle

With the first invocation the target file is set and an empty collection initialized

%tangle [-s] [-f] [-l libfolder] modulepath

With subsequent invocations various commands can be given

%tangle [-r] [-m] [-e] [-f] [labels]

or

%tangle [-a] [libfolder] modulepath

Cell level magic

Cell magic %%entangle updates the code collection

%%entangle [-f] label [dependencies]

Options

The options for %tangle’s first invocation:

• -f, --format: Always reformat cells when inserting or updating
• -l, --lib: changes the root for module paths - also useful for generating tests
• -s, --stdlib: Always add missing stdlib dependencies

For subsequent invocations of %tangle:

• -e, --export: export (tangle) ordered cells to designated file
• -f, --format: reformat the tangled file with autopep8 after export
• -r, --report: report missing dependecies and unused cells
• -m, --missing: list missing dependencies for delayed cells
• -a, --auxiliary: load missing dependencies from module

Note that with -a, unlike the other options the parameters are not labels but the [libfolder] modulepath from the initial creation command is used.

Options for %%entangle are:

• -f, --format: reformat code in cell with autopep8

Required positional arguments:

For the first and the -a invocation of %tangle

• module-path:= The module-path for the export

For %%entangle

• label := The label for a cell - dependencies refer to these

Optional positional arguments:

For the first and the -a invocation of %tangle

• libfolder := A parent folder for the module-path, directly under project-root or project-root/src

For subsequent invocations of %tangle:

• labels := The labels of cells to be exported / reported on

Note that when using labels with --export dependencies are not verified.

For %%entangle

• dependencies:= The labels of dependencies for the code cell

Arguments format

label = name
labels = *1(name *(SPACE name))
dependencies = *1(name *(SPACE name))
modulepath = name *(DOT name)
name = 1*TEXTDATA

Notes

A module-path looks pretty much the same as in a python import statement, the cell-label is a name for the cell within the collection of cells - or python file - specified by the module-path and the dependencies are the labels of other cells within the specified collection upon which this cell depends and which are exported before this cell, if possible.

The code in the cell will only run when the label and dependencies of the cell result in a valid cell sequence. If not, cell execution is delayed until that cell’s dependencies have become consistent again.

Reformatting code can be done for individual cells when updating them, as well as the entire exported file. Reformatting is done by autopep8.

Configuration of autopep8 can be doen through pyproject.toml but relocations of imports - or any code - should be avoided or –auxiliary use will not work properly.

Additional documentation

You can view the documentation local with

quarto preview srcdocs/index.qmd

Or (with a local .venv active) render with

quarto render docs

and use them from from srcdocs/_site

Note: rendering and previewing will also rebuild code in src/