dagstream 0.1.0.dev202310040616

No project description provided

DagStream

DagStream is the Python package in order to manage relationship between functions, especially for data-preprocessing functions for machine learning applications. It offers simple API similar to taskflow.

For more details, please refer to docs below.

• User Manual

How to install

You can intall dagstream by using pip.

pip install dagstream

Basic Usage

Definition of Dag

DagStream class convert your functions into dag nodes.

import dagstream

def funcA():
    print("funcA")

def funcB():
    print("funcB")

def funcC():
    print("funcC")

def funcD():
    print("funcD")

def funcE():
    print("funcE")

def funcF():
    print("funcF")


stream = dagstream.DagStream()
A, B, C, D, E, F = stream.emplace(funcA, funcB, funcC, funcD, funcE, funcF)

A.precede(B, C)
E.succeed(B, C, D)
D.succeed(C)
F.succeed(E)

Execute whole dag functions

dagstream execute the functions based on relationship you defined.

from dagstream.executes import StreamExecutor

# construct functional dag
functional_dag = stream.construct()
executor = StreamExecutor(functional_dag)
executor.run()

In console, follwing items are shown.

funcA
funcC
funcB
funcD
funcE
funcF

Draw relationship of functions using Mermaid

You can draw function dag relationship by using Mermaid.

from dagstream.viewers import MermaidDrawer

functional_dag = stream.construct()
drawer.output(functional_dag, "path/to/output.mmd")

The output is shown like below.

stateDiagram
    direction LR
    state "funcC" as state_0
    state "funcB" as state_1
    state "funcD" as state_2
    state "funcA" as state_3
    state "funcF" as state_4
    state "funcE" as state_5
    [*] --> state_0
    state_0 --> state_1
    state_0 --> state_2
    state_1 --> state_4
    state_2 --> state_3
    state_2 --> state_4
    state_3 --> state_4
    state_4 --> state_5
    state_5 --> [*]