A manage module of parade
Project description
Parade-manage
Parade-manage
is moudle for parade.
Note: You should install parade first.
Install
Install is simple:
> pip install parade-manage
Usage
Tasks dag:
t1 a b c
\ / \ / \ /
d e f
\ / \
g h
Note: t1 is table name, other are task name Enter your project first
> cd your_project
Initialize the class
from parade_manage import ParadeManage
manage = ParadeManage() # or manage = ParadeManage(path='/your/parade/project/path')
If task a is failed, you can get task a's subtask and deps, like
> manage.get_task('a') # get task a
([a, d, e, g, h], {'a': [], 'd': [a], 'e': ['a'], 'g': ['d', 'e'], 'h': [e]})
The first result is tasks, other is deps. And also, you can get mutil failed tasks.
> manage.get_task(['d', 'e']) # get task d and e
(['d', 'e', 'g', 'h'], {'d':[], 'e': [], 'g': ['d', 'e'], 'h': ['e']})
Then, create flow and store flow, generate a yaml file
> manage.store_task_flow('a') # args: 'a' or ['d', 'e']
Or, remove flow by task name(s)
> manage.rm_task_flow('a')
Run the flow, execute failed tasks
manage.run_taskflow('a')
Sometimes, A task does not depend on other tasks, but uses some tables. Now you can get the tables and tasks.
> manage.get_source('d')
['t1', 'a'] # return table 't1' and task name 'a'
And if table t1
is failed, you can use store_source_flow
to store flow
> manage.store_source_flow('t1')
# return a flow related to etl task only
Store DAG to file
>>> manage.store_to_file('analysis.task')
# return a dag
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for parade_manage-0.0.1.10-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9cf7316c6d7c2bf4f17b04ede73720cc95c139c1996f60e41c2a9be7071c3e2d |
|
MD5 | 2d0e85e1bd1be624e2875d65c5d04878 |
|
BLAKE2b-256 | 151de0b13d14451942bd46647fce79b68b63d8128752ec1109a6dddd500c20cb |