ASCII-ART Data Analysis Pipeline Manager
pyleaf is the Python implementation of the Leaf system. The Leaf system is a pipeline (AKA data flow or data analysis protocol) management system that allows to design the pipeline as an ASCII-art diagram through a language called LGL (Leaf Graph Language, see https://github.com/franapoli/lglc).
Main Leaf features include:
- Thin, lightweight, code-independent Abstraction Layer.
- Data flow graph embedded directly into Python source code.
- Automatic creation and management of variables associated with node outputs.
- Automatic persistent storage and retrieval of node outputs.
- Session persistence (i.e. run half project, reboot machine, automatically start again from the last processed node).
- Lazy builds (avoid running nodes that are not necessary for the build of a requested resource).
- Multiprocessing (independent nodes run in parallel).
- Enforcement of code version consistency between nodes (i.e. automatically reprocess all nodes deriving from node A if node A is found to be changed).
- Automatic time and space requirements statistics.
- Automatic publishing (producing hypertext with visual representation of the protocol, processing statistics, link to node outputs, node documentation and source code).