flowrep 0.5.0

flowrep - Your premier tool for workflow representations

flowrep

flowrep — Workflow Recipes from Python

flowrep turns plain Python functions into shareable, versionable workflow recipes — JSON-serialisable "class view" graphs that describe what to compute (which functions, how they connect) without doing the computation or holding any data. Recipes are prospective blueprints that a Workflow Management System (WfMS) can digest, visualise, and execute. To support this, flowrep also provides a set of retrospective, "instance view" data classes for WfMS to populate when executing recipes.

Flowchart-style representations are already the lingua franca for describing processes in science and engineering. flowrep gives you a way to author them in Python and pass them around as simple JSON, validated by Pydantic and enriched with version provenance so they stay robust through time.

Installation

conda install -c conda-forge flowrep   # recommended
pip install flowrep                     # also available from PyPI

Quick Start

Define a couple of simple functions — it will be helpful to make our return statements nicely named variables, but it's not a necessity:

>>> import flowrep as fr

>>> def add(a, b):
...     return a + b

>>> def multiply(x, y):
...     product = x * y
...     return product

Compose them into a workflow with the @workflow decorator. The body reads like normal Python — assign function calls to variables, wire outputs into inputs, return results:

>>> @fr.workflow
... def linear(x, slope, intercept):
...     """y = slope * x + intercept"""
...     scaled = multiply(x, slope)
...     result = add(scaled, intercept)
...     return result

The decorated function is still just its usual callable self:

>>> linear(3, 2, 1)
7

But it now carries a recipe on its flowrep_recipe attribute — a pure-data Pydantic model whose structure you can inspect directly (model_dump(mode="json") gives a Python dict, model_dump_json() gives a JSON string):

>>> recipe = linear.flowrep_recipe.model_dump(mode="json")
>>> recipe["type"]
'workflow'
>>> recipe["inputs"]
['x', 'slope', 'intercept']
>>> recipe["outputs"]
['result']
>>> sorted(recipe["nodes"])
['add_0', 'multiply_0']
>>> recipe["nodes"]["multiply_0"]["outputs"]
['product']
>>> recipe["edges"]
{'add_0.a': 'multiply_0.product'}
>>> recipe["output_edges"]
{'result': 'add_0.output_0'}
>>> recipe["nodes"]["add_0"]["reference"]["info"]["qualname"]
'add'

A key difference between a workflow graph and typical python is that for a graph we need named handles for all of our output values. We can see in the edges that the recipe nicely used our return variable to give the "multiply" a nice output label. For our "add" node, what we returned couldn't be parsed nicely as a label, so in output_edges we see it got assigned a default name "output_0".

Every "atomic" node for running a Python function and every "workflow" not created by parsing a function defition carries a reference recording for the Python function to which it maps (module, qualname, and package version when available), so a WfMS can resolve and execute them later. The recipe also captures exactly how data flows: input_edges wire workflow inputs to child node ports, edges connect sibling outputs to inputs, and output_edges name which child port produces each workflow output.

The recipe is the shareable artifact. It could equally have been authored in a GUI editor or generated by another tool — flowrep doesn't care where it came from, only that it validates. (The availability of referenced functions on your machine, and their actual behaviour, lie outside flowrep's responsibility. We simply check that the recipe is internally consistent.)

Features

Pure Python foundation. Knowing basic Python is enough to get started. The decorators @atomic and @workflow (plus their functional counterparts parse_atomic and parse_workflow) handle the heavy lifting; the recipe format is plain JSON.

Pydantic validation. Recipes are full Pydantic models — they validate on construction, serialise with model_dump_json(), and deserialise with model_validate_json().

Version provenance. Every node reference carries module, qualname, and (optionally) package version metadata. Constraints like forbid_main, forbid_locals, and require_version can be enforced at parse time to ensure recipes only reference published, versioned code.

Composability. Workflows nest: a @workflow-decorated function can call another @workflow-decorated function, and the child's recipe is embedded as a sub-graph. Build complex pipelines from small, testable pieces.

Flow control. Recipes go beyond simple DAGs. The @workflow parser recognises for, while, if/elif/else, and try/except — the same control structures you use in real code. Flow control nodes are inherently dynamic: their exact execution path depends on data and cannot be known until run-time, but their IO signature is always fully known a priori.

Clear separation. Of prospective, "class view" recipes, and retrospective, "instance view" data objects.

Example: flow control and nesting

So far we've seen "workflow" nodes, and alluded to "atomic" nodes. You can also explicitly attach an atomic recipe to a function definition, but most of the time you won't need to since the workflow parser auto-parses undecorated function calls as atomic nodes.

On the other hand, we can parse @workflow-decorated functions as workflows to compose complex workflows with nested subgraphs.

Beyond simple "atomic" and DAG "workflow" nodes, there are also recipe formats for flow control structures like while. Just like the other nodes, these can be written directly as JSON, or parsed inside a workflow from a python function definition -- with some syntax restrictions. This is covered in more detail in the user guide, but here let's see both of these features in use at once by nesting a while loop inside a sub-workflow. Let's look at a while loop, where our main syntax restriction is that the condition must be a function call:

>>> def is_less_than_target(value, target):
...     result = value < target
...     return result

>>> @fr.atomic  # This is optional, but doesn't hurt us
... def double(x):
...     doubled = x * 2
...     return doubled

>>> @fr.workflow
... def double_until(x, target):
...     """Repeatedly double `x` until it reaches `target`."""
...     while is_less_than_target(x, target):
...         x = double(x)
...     return x

>>> double_until(3, 40)
48

And then we can compose workflows by nesting:

>>> @fr.workflow
... def double_and_add(a, b, target):
...     big_a = double_until(a, target)
...     result = add(big_a, b)
...     return result

>>> double_and_add(3, 100, 40)
148

The resulting recipe captures the full structure — the while-loop, the nested workflow, and all the edges between them — as a single pydantic model that can be dumped to a JSON document.

We can see the layers of nested subgraphs:

>>> recipe_json = double_and_add.flowrep_recipe.model_dump(mode="json")
>>> [child for child in recipe_json["nodes"]]
['double_until_0', 'add_0']

>>> [nested_child for nested_child in recipe_json["nodes"]["double_until_0"]["nodes"]]
['while_0']

Although the while-node is dynamic -- and therefore we can't know its exact nodes until run-time, it must and does still have well-defined IO signature. We can look at the template it will follow, e.g., by peeking at the part of the recipe used for the "while" condition:

>>> recipe_json["nodes"]["double_until_0"]["nodes"]["while_0"]["case"]["condition"]["node"]["type"]
'atomic'

>>> recipe_json["nodes"]["double_until_0"]["nodes"]["while_0"]["case"]["condition"]["node"]["reference"]["info"]["qualname"]
'is_less_than_target'

And to see how it will forward it's inputs down into its prospective subgraph:

>>> recipe_json["nodes"]["double_until_0"]["nodes"]["while_0"]["input_edges"]
{'condition.value': 'x', 'condition.target': 'target', 'body.x': 'x'}

We run these in the examples above to show two things: first, that even when nested, the decorated functions are still just python functions; second, to show in the following section that the recipe we parse from this are intended to give the same result as these underlying functions when we run the recipe with a WfMS.

Beyond Recipes: Retrospective "Instance View" Data and Execution

Recipes are prospective — they describe a computation template without holding data. For retrospective analysis (inspecting what actually happened during a run), flowrep provides two additional layers accessible through the API:

>>> from flowrep.api import tools as frt

flowrep.api.tools.recipe2data converts a recipe from the prospective "class view" to the retrospective "instance view" — a mutable data structure whose input and output ports can hold actual Python values. Retrospective data objects mirror the recipe graph but trade JSON-serializability for the ability to carry arbitrary data:

>>> wf_data = frt.recipe2data(double_and_add.flowrep_recipe)

flowrep.api.tools.run_recipe goes one step further: it executes the recipe with the provided inputs and returns a fully populated data object. This is powered by a minimal, built-in WfMS intended as a reference implementation and for use in tests and documentation (like this!):

>>> retrospective = frt.run_recipe(
...     double_and_add.flowrep_recipe, a=3, b=100, target=40
... )
>>> retrospective.output_ports["result"].value
148

Because every child node's ports are populated too, the retrospective data graph gives you full data provenance — you can walk the tree and inspect exactly what each node received and produced. For flow control nodes, which are prospectively "black boxes", we find that retrospectively they are simple DAGs. For our while-loop, that means that we can retrospectively inspect the provenance of each loop iteration:

>>> while_loop = retrospective.nodes["double_until_0"].nodes["while_0"]
>>> while_loop.nodes["body_0"].output_ports["x"].value
6
>>> while_loop.nodes["body_1"].output_ports["x"].value
12

For a deeper look at all available node types, edge semantics, version provenance, and the retropective data/WfMS layer, see the user guide.

Documentation

• The user guide notebook comprehensively covers all node types, edge models, flow control, versioning, retrospective data formats, a demo WfMS implementation, and recipe format converters. Launch it interactively on mybinder.
• readthedocs

Contributing

Contributions are welcome! Please see CODE_OF_CONDUCT.md for community guidelines.