Skip to main content

No project description provided

Project description

DagStream

DagStream is the Python package in order to manage relationship between functions, especially for data-preprocessing functions for machine learning applications. It offers simple API similar to taskflow.

For more details, please refer to docs below.

How to install

You can intall dagstream by using pip.

pip install dagstream

Basic Usage

Definition of Dag

DagStream class convert your functions into dag nodes.

import dagstream

def funcA():
    print("funcA")

def funcB():
    print("funcB")

def funcC():
    print("funcC")

def funcD():
    print("funcD")

def funcE():
    print("funcE")

def funcF():
    print("funcF")


stream = dagstream.DagStream()
A, B, C, D, E, F = stream.emplace(funcA, funcB, funcC, funcD, funcE, funcF)

A.precede(B, C)
E.succeed(B, C, D)
D.succeed(C)
F.succeed(E)

Execute whole dag functions

dagstream execute the functions based on relationship you defined.

from dagstream.executor import StreamExecutor

# construct functional dag
functional_dag = stream.construct()
executor = StreamExecutor(functional_dag)
executor.run()

In console, follwing items are shown.

funcA
funcC
funcB
funcD
funcE
funcF

Draw relationship of functions using Mermaid

You can draw function dag relationship by using Mermaid.

from dagstream.viewers import MermaidDrawer

functional_dag = stream.construct()
drawer.output(functional_dag, "path/to/output.mmd")

The output is shown like below.

stateDiagram
    direction LR
    state "funcE" as state_0
    state "funcB" as state_1
    state "funcF" as state_2
    state "funcD" as state_3
    state "funcC" as state_4
    state "funcA" as state_5
    state_0 --> state_2
    state_1 --> state_0
    state_2 --> [*]
    state_3 --> state_0
    state_4 --> state_0
    state_4 --> state_3
    [*] --> state_5
    state_5 --> state_4
    state_5 --> state_1

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

dagstream-0.1.5-py3-none-any.whl (13.8 kB view details)

Uploaded Python 3

File details

Details for the file dagstream-0.1.5-py3-none-any.whl.

File metadata

  • Download URL: dagstream-0.1.5-py3-none-any.whl
  • Upload date:
  • Size: 13.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.9.19 Linux/5.15.0-106-generic

File hashes

Hashes for dagstream-0.1.5-py3-none-any.whl
Algorithm Hash digest
SHA256 157dcdc62567517ea45d49d7d1e6cf29d3cb719785b5409345c8b6fcf2c5af56
MD5 638e91b509187a9ccddc10f4a3bed38a
BLAKE2b-256 d9d558be819db71d05172abe78d0f83940efe6bf13e89fcbb2d8f96e24a2338c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page