Skip to main content

Make computer lazy and build a dynamic task graph for your project!

Project description

Task-Graph

Build a dynamic task graph for your project with Task-Graph!

Introduction

When facing a complex task, it's often necessary to recalculate the entire task when user wants to get one of the intermediate results, or to adjust one of the parameters or methods, but once you have Task-Graph,

  1. If you need to get a result, Task-Graph will only compute all the upstream tasks of the result, and each task will cache for later use.

  2. If one of the task's parameters or method is adjusted, Task-Graph will automatically check all the downstream tasks and clear their cached results.

In short, Task-Graph will make the computer lazy and just fulfill your needs without doing repetitive tasks, no more or less.

Usage

No doc available now, here is an minimal example.

# import
from task_graph import TaskGraph

# init a TaskGraph
tg = TaskGraph()

# user task methods
def add(a, b):
    print("add", a, b)
    return a + b


def sub(a, b):
    print("sub", a, b)
    return a - b

# describe the task graph(much like dask.delayed)
ret1 = tg.add_task(add)(1, 2)
ret2 = tg.add_task(add)(3, ret1)
ret3 = tg.add_task(add)(4, ret2)
ret4 = tg.add_task(sub)(ret2, ret3)
final = tg.add_task('to_list')(ret1, ret2, ret3, ret4)

# trigger the computation and print
print(final.compute())

# when I want to update a task
tg.update_task(ret3)(add)(5, ret2)

# Because ret3 is change, rest3, rest4 and final will be recalculated
print(final.compute())

Or, a lazy way:

from task_graph import TaskGraph


tg = TaskGraph()

def add(a, b):
    print("add", a, b)
    return a + b


def sub(a, b):
    print("sub", a, b)
    return a - b

# omit add_task
ret1 = tg(add)(1, 2)
ret2 = tg(add)(3, ret1)
ret3 = tg(add)(4, ret2)
ret4 = tg(sub)(ret2, ret3)
final = tg('to_list')(ret1, ret2, ret3, ret4)

# auto compute and print
final.print()

# omit update_task
tg(ret3)(add)(5, ret2)

# auto compute and print
final.print()

TODO

Task-Graph is at the very beginning, so many todos in the codes. Here are a few general ones:

  • <input type="checkbox" disabled="" /> add parallel scheduler
  • <input type="checkbox" disabled="" /> add decorator that can auto transfer function to task
  • <input type="checkbox" disabled="" /> add graphviz-based visualization
  • <input type="checkbox" disabled="" /> add unit test
  • <input type="checkbox" disabled="" /> add doc
  • <input type="checkbox" disabled="" /> add examples

Zen

I am lazy, so I build this to let computer be also lazy.

One of the biggest differences in Python from other languages is that the methods (like all other objects) are mutable, Task-Graph is designed to accommodate that.

Task-Graph was designed to speed up python project, and want to make it the simplest solution to avoid any recalculation.

I have referred to the Dask.delayed API in many places. The difference between Task-Graph and Dask is to try to adapt to the dynamics of the python and python projects, and make computer not only delayed but also lazy.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for task-graph, version 0.0.1
Filename, size File type Python version Upload date Hashes
Filename, size task_graph-0.0.1-py3-none-any.whl (7.8 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size task-graph-0.0.1.tar.gz (5.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page