illumo-flow
Workflow orchestration primitives featuring declarative DSL wiring, routing control, and fail-fast execution.
Installation
pip install illumo-flow
Quick Example
from illumo_flow import Flow, FunctionNode
# Define lightweight callables (payload-first; shared context access is opt-in)
def extract(payload):
return {"customer_id": 42, "source": "demo"}
def transform(payload):
return {**payload, "normalized": True}
def load(payload):
return f"stored:{payload['customer_id']}"
nodes = {
"extract": FunctionNode(extract, name="extract", outputs="$ctx.data.raw"),
"transform": FunctionNode(
transform,
name="transform",
inputs="$ctx.data.raw",
outputs="$ctx.data.normalized",
),
"load": FunctionNode(
load,
name="load",
inputs="$ctx.data.normalized",
outputs="$ctx.data.persisted",
),
}
flow = Flow.from_dsl(
nodes=nodes,
entry="extract",
edges=["extract >> transform", "transform >> load"],
)
context = {}
flow.run(context)
print(context["data"]["persisted"]) # stored:42
# Flow.run returns the mutated context; per-node outputs remain
# available under `context["payloads"]`.
Examples & CLI
The GitHub repository ships reference examples and a CLI (e.g. python -m examples linear_etl).
Clone the repo if you want to explore them locally:
git clone https://github.com/kitfactory/illumo-flow.git
cd illumo-flow
python -m examples linear_etl
YAML Configuration
Flows can also be defined in configuration files:
flow:
entry: extract
nodes:
extract:
type: illumo_flow.core.FunctionNode
name: extract
context:
inputs:
callable: examples.ops.extract
outputs: $ctx.data.raw
transform:
type: illumo_flow.core.FunctionNode
name: transform
context:
inputs:
callable: examples.ops.transform
payload: $ctx.data.raw
outputs: $ctx.data.normalized
load:
type: illumo_flow.core.FunctionNode
name: load
context:
inputs:
callable: examples.ops.load
payload: $ctx.data.normalized
outputs: $ctx.data.persisted
edges:
- extract >> transform
- transform >> load
context.inputs.callable supplies the Python callable path for each node. Literal strings are imported at build time, while expressions (e.g. $ctx.registry.my_func) are evaluated during execution.
Expressions
$ctx.*accesses the shared context (e.g.$ctx.data.raw). Writingctx.*or the shorthand$.pathis automatically normalized to the same form.$payload.*reads fromcontext["payloads"]$joins.*reads fromcontext["joins"]$env.VARreads environment variables- Template strings like
"Hello {{ $ctx.user.name }}"are supported ininputsdefinitions
from illumo_flow import Flow
flow = Flow.from_config("./flow.yaml")
context = {}
flow.run(context)
print(context["data"]["persisted"])
Payload vs Context
- Flow resolves each node's
payloadfrom the declaredinputs. - Nodes return the next
payload; Flow stores it undercontext["payloads"][node_id]and writes to the paths declared inoutputs. - Treat the payload as the primary contract. Only nodes created with
allow_context_access=Truecan reach the shared dictionary (e.g., for metrics viacontext.setdefault("metrics", {})); everyone else operates purely on payloads.
Branching
- To route dynamically, return a dictionary mapping successor identifiers to payloads (e.g.
{"approve": payload}). Only the listed successors are executed. - Returning an empty dictionary
{}stops downstream execution; returning multiple keys broadcasts to all corresponding successors.
Testing (repository clone)
Keep runs short and deterministic by executing one test at a time.
- Update or extend scenarios inside
tests/test_flow_examples.py(edit-only policy for this repo). - Execute
pytest tests/test_flow_examples.py::TEST_NAME; setFLOW_DEBUG_MAX_STEPS=200when exercising looping flows to guard against hangs. - Track progress in
docs/test_checklist.mdand reset all checkboxes before regression sweeps.
Refer to docs/test_checklist.md for the live checklist.
Documentation
- Architecture and API design: docs/flow.md
- Japanese version: docs/flow_ja.md
- Concepts overview: docs/concept.md
- Quick tutorial reference: docs/tutorial.md / docs/tutorial_ja.md
- Chaptered tutorial (design foundations + samples): docs/tutorials/README.md / docs/tutorials/README_ja.md
Highlights
- DSL edges such as
A >> B,(A & B) >> C - Payload-first callable interface(
allow_context_access=Trueを指定したノードのみ共有コンテキストへ明示アクセス) - LoopNode for per-item iteration (self edge
loop >> loop+ body routeloop >> worker) - Branching via returned mappings (e.g.
{successor: payload}) - Built-in join handling (nodes with multiple parents automatically wait for all inputs)
- Examples covering ETL, dynamic routing, fan-out/fan-in, timeout handling, and early stop
Metadata
Release files for illumo-flow 0.1.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| illumo_flow-0.1.4.tar.gz | 17.5 kB | Without provenance | Details for illumo_flow-0.1.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| illumo_flow-0.1.4-py3-none-any.whl | Python 3 | none | any | Details for illumo_flow-0.1.4-py3-none-any.whl |
Total release size: 31.2 kB
Release files / illumo_flow-0.1.4.tar.gz
| Download URL | illumo_flow-0.1.4.tar.gz |
|---|---|
| Size | 17.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
53c1893ceeacac7f5b7c2803fe9c7796aba83c57048b82edd1fe107fa96e2c9f
|
|
BLAKE2b-256 checksum How to use checksums |
d5611cc4c3a8b0a4732ce05f734d8deae6520cb82f4f81d6ecfa7d6d1d48272e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.12
|
Release files / illumo_flow-0.1.4-py3-none-any.whl
| Download URL | illumo_flow-0.1.4-py3-none-any.whl |
|---|---|
| Size | 13.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c063362471c29991491e5ffb05076635825b77c11a1057606ef6e4cf73033878
|
|
BLAKE2b-256 checksum How to use checksums |
d37255add85cd554ef888b50329b5022ccf619fdd9523e5587e99b154286e210
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.12
|
Security for illumo-flow 0.1.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.