A manage module of parade
Project description
Parade-manage
Parade-manage
is moudle for parade.
Note: You should install parade first.
Install
Install is simple:
> pip install parade-manage
Usage
Tasks dag:
t1 a b c
\ / \ / \ /
d e f
\ / \
g h
Note: t1 is table name, other are task name Enter your project first
> cd your_project
Initialize the class
from parade_manage import ParadeManage
manage = ParadeManage() # or manage = ParadeManage(path='/your/parade/project/path')
If task a is failed, you can get task a's subtask and deps, like
> manage.get_task('a') # get task a
([a, d, e, g, h], {'a': [], 'd': [a], 'e': ['a'], 'g': ['d', 'e'], 'h': [e]})
The first result is tasks, other is deps. And also, you can get mutil failed tasks.
> manage.get_task(['d', 'e']) # get task d and e
(['d', 'e', 'g', 'h'], {'d':[], 'e': [], 'g': ['d', 'e'], 'h': ['e']})
Then, create flow and store flow, generate a yaml file
> manage.store_task_flow('a') # args: 'a' or ['d', 'e']
Or, remove flow by task name(s)
> manage.rm_task_flow('a')
Run the flow, execute failed tasks
manage.run_taskflow('a')
Sometimes, A task does not depend on other tasks, but uses some tables. Now you can get the tables and tasks.
> manage.get_source('d')
['t1', 'a'] # return table 't1' and task name 'a'
And if table t1
is failed, you can use store_source_flow
to store flow
> manage.store_source_flow('t1')
# return a flow related to etl task only
Store DAG to file
>>> manage.store_to_file('analysis.task')
# return a dag
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.