tx-parallex 0.0.22

A job queue with data dependencies

parallex

Introduction

A queue with dependencies

Usage

from parallex import run

ret = run(number_of_workers = 4, specf = "spec.yml", dataf = "data.yml")

Spec

Each task is given a dict called data.

let

The let task sets data

type: let
obj: 
  <var>: <value>
  ...
  <var>: <value>
sub: <subtask>

map

The map task reads a list coll from data and applies a list of subtasks to each member of the list. The members will be assigned to var in data passed to those tasks

type: map
coll: <variable name for collection>
var: <variable name>
sub: <subtask>

top

The top task toplogically sorts subtasks.

type: top
sub: <subtasks>

It reads the depends_on property of subtasks, which has format:

<task name>: [<param>, ..., <param>]
...
<task name>: [<param>, ..., <param>]

The result of a task will be assigned to the parameters that it maps to.

python

The python task runs a python function. It reads parameters from data.

type: python
name: <name>
mod: <module>
func: <function>
params: <parameters>
depends_on: <dependencies>

params are the same format as depends_on

Data

data can be arbitrary yaml