qitangle 0.5.5

QiTangle - IPython magic to tangle (quarto) code cells into python source files

QiTangle - IPython magic to entangle code from notebooks.

export code from notebook cells to python source files

The first Quarto document I created was used to implement a python module based on some math formula’s from an article (about measuring sap flux in tree stems). Quarto let me follow the article and methodically, step by step functions were build and documented along the formula’s and some example calculations. This worked like a charm. When everything was implemented and working all there was left to do was copy the code from the document into a separate python file for use in our runtime environment.

Unfortunately, any changes in eihter of the two sources - the notebook and the script - had to be copied from one to theother. What I needed was a way to convert or export selected code cells into a python source code file automatically and I found this in NBDev. NBDev exports your code, builds the documentation and, if desired, packages the code and uploads to PyPi.

Jupyter notebooks however are a uncomfortable to use in combination with git, even with the clean-up tools provided by NBDev, many unneeded git diffs were checked in and worse; merge conflicts can be a huge problem.

Something closer to using Quarto’s own markdown was needed, simply extracting or exporting python sourcefiles from Quarto documents. Quarto’s python support is based on IPython and Jupyter kernels, and these offer an extension mechanism called magics. These extensions are available in Jupyter and Quarto notebooks.

Not finding what I needed I wrote a magic to export code from notebook cells to python source files and called it QiTangle after Quarto, IPython and TANGLE, the latter being the name of a component and process in Knuth’s Literate programming system that was responsible for tangling the computer code together.

QiTangle and literate programming

Quarto’s rich document creation and interactive computing extended with the ability to export selected Python code cells into sourcecode file are not entirely dissimilar to Knuth’s Literate programming paradigm. WEB’s system allowed for describring pieces of computer code and algorithms that could be combined with each other. QiTangle is more restricted in that it only allows chunks of lines of code to be exported to a python source file, without combining expressions. It does however, unlike many similar notebook based systems, allow cells to be reordered into the exported source file. As in Knuth’s system, QiTangle allows disconnecting the narrative order that descibes the code in an order suitable to explain it, from the neccesities imposed on the ordering of code by the computer language.

Installing

pip install qitangle

Usage

Loading the extension

import qitangle
%reload_ext qitangle.entangle

Using %reload_ext will reset the cell collection even when a kernel process remains active as is the case with Quarto

Tangling

Tangling is the process of exporting the individual code cells to a file in the order as specified by the label and dependencies attributes of an entangled cell.

Each %%entangle invocation adds a cell to a collection, it’s label and dependencies inform the %tangle which cells depend on which other cells and thereby ultimately the order in which they are exported. By exporting it’s dependencies before the cell we ensure that prior to excution of that cell everything, well it’s code dependencies at least, are in place.

When updating results in a collection that can not be ordered correctly cell execution is delayed until their dependencies are added.

Optionally each cell individually can be reformatted through autopep8 and optionally when exporting, the entire produced file can be reformatted. Configuration of autopep8 is described in it’s manual.

Invocations

Line level magic

Line magic %tangle controls the tangle

With the first invocation the target file is set

%tangle [-l libfolder] modulepath

With subsequent invocations various commands are given

%tangle [-r] [-e] [-m] [-s] [--stdlib] [labels]

Cell level magic

Cell magic %%entangle updates the code collection

%%entangle [-f] label [dependencies]

Options

The options for %tangle’s first invocation:

• -f, --format: Always reformat the tangled file with autopep8 after export
• -l, --lib: changes the root for module paths - also useful for generating tests

For subsequent invocations of %tangle:

• -e, --export: export ordered cells to designated file
• -f, --format: reformat the tangled file with autopep8 after export
• -r, --report: report missing dependecies and unused cells
• -m, --missing: list missing dependencies for delayed cells
• -s, --stdlib: adds all currently missing stdlib dependencies

Options for %%entangle are:

• -f, --format: reformat code in cell with autopep8

Required positional arguments:

For the first invocation of %tangle

• module-path:= The module-path for the export

For %%entangle

• label := The label for a cell - dependencies refer to these

Optional positional arguments:

For the first invocation of %tangle

• libfolder := A parent folder for the module-path, directly under project-root or project-root/src

For subsequent invocations of %tangle:

• labels := The labels of cells to be exported / reported on

Note that when using labels with --export dependencies are not verified.

For %%entangle

• dependencies:= The labels of dependencies for the code cell

Argument format

label = name
labels = *1(name *(SPACE name))
dependencies = *1(name *(SPACE name))
modulepath = name *(DOT name)
name = 1*TEXTDATA

Discussion

A module-path looks pretty much the same as in a python import statement, the cell-label is a name for the cell within the collection of cells - or python file - specified by the module-path and the dependencies are the labels of other cells within the specified collection upon which this cell depends and which are exported before this cell, if possible.

When an inconsistency is found in the dependency information the export will fail with a warning and nothing will be (over-) written. Fixing the consistency in the dependicies is the only remedy.

The code in the cell will only run when the label and dependencies of the cell result in a valid code sequence. If not, cell execution is delayed until that cell’s dependencies have become consistent again.

Reformatting code can be done for individual cells when updating them, as well as the entire exported file. Reformatting is done by autopep8.

Additional documentation

You can view the documentation local with

quarto preview srcdocs/index.qmd

Or (with a local .venv active) render with

quarto render docs

and use them from from srcdocs/_site

Note: rendering and previewing will also rebuild code in src/