A manage module of parade
Project description
Parade-manage
Parade-manage
is moudle for parade.
Note: You should install parade first.
Install
Install is simple:
> pip install parade-manage
Usage
Tasks dag:
t1 a b c
\ / \ / \ /
d e f
\ / \
g h
Note: t1 is table name, other are task name Enter your project first
> cd your_project
Initialize the class
from parade_manage import ParadeManage
manage = ParadeManage() # or manage = ParadeManage(path='/your/parade/project/path')
If task a is failed, you can get task a's subtask and deps, like
> manage.get_task('a') # get task a
([a, d, e, g, h], {'a': [], 'd': [a], 'e': ['a'], 'g': ['d', 'e'], 'h': [e]})
The first result is tasks, other is deps. And also, you can get mutil failed tasks.
> manage.get_task(['d', 'e']) # get task d and e
(['d', 'e', 'g', 'h'], {'d':[], 'e': [], 'g': ['d', 'e'], 'h': ['e']})
Then, create flow and store flow, generate a yaml file
> manage.store_task_flow('a') # args: 'a' or ['d', 'e']
Or, remove flow by task name(s)
> manage.rm_task_flow('a')
Run the flow, execute failed tasks
manage.run_taskflow('a')
Sometimes, A task does not depend on other tasks, but uses some tables. Now you can get the tables and tasks.
> manage.get_source('d')
['t1', 'a'] # return table 't1' and task name 'a'
And if table t1
is failed, you can use store_source_flow
to store flow
> manage.store_source_flow('t1')
# return a flow related to etl task only
Store DAG to file
>>> manage.store_to_file('analysis.task')
# return a dag
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for parade_manage-0.0.1.12-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f9d87a4b5aaac4b14d76118e980eb21ac2cf554783af26dcb805aa931ee3541 |
|
MD5 | d90e3d12eb561b3541c204aa633191b6 |
|
BLAKE2b-256 | ca7ed2d23caad9c2efcc1ed06e29585f531283f91076d942905e310d057a4888 |