qitangle 0.5.3

QiTangle - IPython magic to tangle (quarto) code cells into python source files

QiTangle - IPython magic to entangle code from notebooks.

export code from notebook cells to python source files

The first Quarto document I created was used to implement a python module based on some math formula’s from an article (about measuring sap flux in tree stems). Quarto let me follow the article and methodically, step by step functions were build and documented along the formula’s and some example calculations. This worked like a charm. When everything was implemented and working all there was left to do was copy the code from the document into a separate python file for use in our runtime environment.

Unfortunately, any changes in eihter of the two sources - the notebook and the script - had to be copied from one to theother. What I needed was a way to convert or export selected code cells into a python source code file automatically and I found this in NBDev. NBDev exports your code, builds the documentation and, if desired, packages the code and uploads to PyPi.

Jupyter notebooks however are a uncomfortable to use in combination with git, even with the clean-up tools provided by NBDev, many unneeded git diffs were checked in and worse; merge conflicts can be a huge problem.

Something closer to using Quarto’s own markdown was needed, simply extracting or exporting python sourcefiles from Quarto documents. Quarto’s python support is based on IPython and Jupyter kernels, and these offer an extension mechanism called magics. These extensions are available in Jupyter and Quarto notebooks.

Not finding what I needed I wrote a magic to export code from notebook cells to python source files and called it QiTangle after Quarto, IPython and TANGLE, the latter being the name of a component and process in Knuth’s Literate programming system that was responsible for tangling the computer code together.

QiTangle and literate programming

Quarto’s rich document creation and interactive computing extended with the ability to export selected Python code cells into sourcecode file are not entirely dissimilar to Knuth’s Literate programming paradigm. WEB’s system allowed for describring pieces of computer code and algorithms that could be combined with each other. QiTangle is more restricted in that it only allows chunks of lines of code to be exported to a python source file, without combining expressions. It does however, unlike many similar notebook based systems, allow cells to be reordered into the exported source file. As in Knuth’s system, QiTangle allows disconnecting the narrative order that descibes the code in an order suitable to explain it, from the neccesities imposed on the ordering of code by the computer language.

Installing

pip install qitangle

Usage

Loading the extension

import qitangle
%reload_ext qitangle.entangle

Using %reload_ext will reset the cell collection even when a kernel process remains active as is the case with Quarto

Tangling

Tangling is the process of exporting the individual code cells to a file in the order as specified by the label and dependencies attributes of an entangled cell.

Each entangle invocation adds a cell to a collection, the label and dependencies determine which cells depend on which and thus ultimately the order in which they are exported by exporting the dependencies before the code cell. When updating results in a collection that can not be ordered correctly a warning will be given and cell execution is delayed until the collection becomes valid again. As long as the collection sequence is not valid no file will be written.

Optionally each cell can be reformatted through autopep8. When exporting the entire produced file can be reformatted. Configuration of autopep8 is described in it’s manual.

Invocation

Line level magic

Line magic basically controls the tangle

Fist invocation

%tangle [-l libfolder] modulepath

Following invocations

%tangle [-r] [-e] [labels]

Cell level magic

Cell magic updates the code collection

%%entangle [-f] label [dependencies]

Options

• -e, --export: export ordered cells to designated file
• -r, --report: report missing dependecies and unused cells
• -f, --fix: Fix/reFormat code with autopep8
• -l, --lib: changes the root for module paths - also useful for generating tests

Required positional arguments:

• module-path:= The module-path for the export
• label := The label for a cell - dependencies refer to these

Optional positional argument:

• libfolder := A parent folder for the module-path, directly under project-root or project-root/src
• dependencies:= The labels of dependencies for the code cell
• labels := The labels of cells and depencies to be exported / reported on

Argument format

label = name
labels = *1(name *(SPACE name))
dependencies = *1(name *(SPACE name))
modulepath = name *(DOT name)
name = 1*TEXTDATA

Discussion

A module-path looks pretty much the same as in a python import statement, the cell-label is a name for the cell within the collection of cells - or python file - specified by the module-path and the dependencies are the labels of other cells within the specified collection upon which this cell depends and which are exported before this cell, if possible.

When an inconsistency is found in the dependency information the export will fail with a warning and nothing will be (over-) written. Fixing the consistency in the dependicies is the only remedy.

The code in the cell will only run when the label and dependencies of the cell result in a valid code sequence. If not, cell execution is delayed until that cell’s dependencies have become consistent again.

Reformatting code can be done for individual cells when updating them, as well as the entire exported file. Reformatting is done by autopep8.

Additional documentation

You can view the documentation local with

quarto preview srcdocs/index.qmd

Or (with a local .venv active) render with

quarto render docs

and use them from from srcdocs/_site

Note: rendering and previewing will also rebuild code in src/