A job queue with data dependencies
Project description
parallex
Introduction
A queue with dependencies
Usage
from parallex import run
ret = run(number_of_workers = 4, specf = "spec.yml", dataf = "data.yml")
Spec
Each task is given a dict called data
.
let
The let
task sets data
type: let
obj:
<var>: <value>
...
<var>: <value>
sub: <subtask>
map
The map
task reads a list coll
from data
and applies a list of subtasks to each member of the list. The members will be assigned to var
in data
passed to those tasks
type: map
coll: <variable name for collection>
var: <variable name>
sub: <subtask>
top
The top
task toplogically sorts subtasks.
type: top
sub: <subtasks>
It reads the depends_on
property of subtasks, which has format:
<task name>: [<param>, ..., <param>]
...
<task name>: [<param>, ..., <param>]
The result of a task will be assigned to the parameters that it maps to.
python
The python
task runs a python function. It reads parameters from data
.
type: python
name: <name>
mod: <module>
func: <function>
params: <parameters>
depends_on: <dependencies>
ret: <returns>
params
are the same format as depends_on
dsl
A dsl block contains a subset of python.
type: dsl
python: <python>
Available syntax:
assignment
<var> = <const> | <list> | <dict>
This translates to let
.
function application
<var> = <module>.<func>(<param>=<arg>, ...)
This translate to python
.
where <var>
is name
return
return <dict>
this translates to ret
in python
.
Data
data can be arbitrary yaml
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tx_parallex-0.0.27-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b6ebf68c7f5d3d8de9fc17e724f6efa5522c9281b0793a051e08fae0f8c230b |
|
MD5 | 87532ca05dbd4c612973da6b3e3c38c6 |
|
BLAKE2b-256 | c345d3ce423fc9dcbb0f1dd92764a1141b3e741af914137beef047dece58f03c |