Skip to main content

A manage module of parade

Project description

Parade-manage

Parade-manage is moudle for parade.

Note: You should install parade first.

Install

Install is simple:

> pip install parade-manage

Usage

Tasks dag:

 t1  a   b   c
  \ / \ / \ / 
   d   e   f
    \ / \
     g   h

Note: t1 is table name, other are task name Enter your project first

> cd your_project

Initialize the class

from parade_manage import ParadeManage
	
manage = ParadeManage() # or manage = ParadeManage(path='/your/parade/project/path')

If task a is failed, you can get task a's subtask and deps, like

> manage.get_task('a') # get task a
([a, d, e, g, h], {'a': [], 'd': [a], 'e': ['a'], 'g': ['d', 'e'], 'h': [e]})

The first result is tasks, other is deps. And also, you can get mutil failed tasks.

> manage.get_task(['d', 'e']) # get task d and e
(['d', 'e', 'g', 'h'], {'d':[], 'e': [], 'g': ['d', 'e'], 'h': ['e']})

Then, create flow and store flow, generate a yaml file

> manage.store_task_flow('a') # args: 'a' or ['d', 'e']  

Or, remove flow by task name(s)

> manage.rm_task_flow('a')

Run the flow, execute failed tasks

manage.run_taskflow('a')

Sometimes, A task does not depend on other tasks, but uses some tables. Now you can get the tables and tasks.

> manage.get_source('d')
['t1', 'a']  # return table 't1' and task name 'a'

And if table t1 is failed, you can use store_source_flow to store flow

> manage.store_source_flow('t1')
# return a flow related to etl task only

Store DAG to file

>>> manage.store_to_file('analysis.task')
# return a dag

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for parade-manage, version 0.0.1.8
Filename, size File type Python version Upload date Hashes
Filename, size parade-manage-0.0.1.8.tar.gz (5.6 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page