Parallel task graph framework.
Project description
About TaskGraph
TaskGraph is great.
TaskGraph Dependencies
Task Graph is written in pure Python, but if the psutils package is installed the distributed multiprocessing processes will be niced.
Example Use
Install taskgraph with
pip install taskgraph
Then
def _create_list_on_disk(value, length, target_path):
"""Create a numpy array on disk filled with value of `size`."""
target_list = [value] * length
pickle.dump(target_list, open(target_path, 'wb'))
def _sum_lists_from_disk(list_a_path, list_b_path, target_path):
"""Read two lists, add them and save result."""
list_a = pickle.load(open(list_a_path, 'rb'))
list_b = pickle.load(open(list_b_path, 'rb'))
target_list = []
for a, b in zip(list_a, list_b):
target_list.append(a+b)
pickle.dump(target_list, open(target_path, 'wb'))
# create a taskgraph that uses 4 multiprocessing subprocesses when possible
task_graph = taskgraph.TaskGraph(self.workspace_dir, 4)
target_a_path = os.path.join(self.workspace_dir, 'a.dat')
target_b_path = os.path.join(self.workspace_dir, 'b.dat')
result_path = os.path.join(self.workspace_dir, 'result.dat')
result_2_path = os.path.join(self.workspace_dir, 'result2.dat')
value_a = 5
value_b = 10
list_len = 10
task_a = task_graph.add_task(
target=_create_list_on_disk,
args=(value_a, list_len, target_a_path),
target_path_list=[target_a_path])
task_b = task_graph.add_task(
target=_create_list_on_disk,
args=(value_b, list_len, target_b_path),
target_path_list=[target_b_path])
sum_task = task_graph.add_task(
target=_sum_lists_from_disk,
args=(target_a_path, target_b_path, result_path),
target_path_list=[result_path],
dependent_task_list=[task_a, task_b])
sum_task.join()
# expect that result is a list `list_len` long with `value_a+value_b` in it
result = pickle.load(open(result_path, 'rb'))
Running Tests
Taskgraph includes a tox configuration for automating builds across multiple python versions and whether psutil is installed. To execute all tests, run:
$ tox
Alternatively, if you’re only trying to run tests on a single configuration (say, python 3.5 without psutil), you’d run:
$ tox -e py35-base
Or if you’d like to run the tests for the combination of Python 2.7 with psutil, you’d run:
$ tox -e py27-psutil
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
taskgraph-0.5.2.tar.gz
(18.0 kB
view hashes)
Built Distributions
taskgraph-0.5.2-py3-none-any.whl
(11.6 kB
view hashes)
taskgraph-0.5.2-py2-none-any.whl
(13.5 kB
view hashes)
Close
Hashes for taskgraph-0.5.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b260c8975038ed580d94cef7a863fe39ed0f6a8cb164fb1ed42fcb8691172ffb |
|
MD5 | fb90cb85bbd4b4197458b34f98743761 |
|
BLAKE2b-256 | 14d6f78a21e6d4cf0c9736993e4c7354a03f88dc538261caec4a4dc4601b1a2b |
Close
Hashes for taskgraph-0.5.2-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01f2d46ee21f4bec41204259806123a63aa2381742cda20aaebd1f4366f01852 |
|
MD5 | d7273e139fe23da78aa29f0ad75c7ff9 |
|
BLAKE2b-256 | eba8b299cde0c3f18c128c55a5d106c18d6396b8eb2d3cd5f587b21f78572a5b |