Package extends Airflow functionality with CWL v1.0 support
Project description
cwl-airflow-parser
About
Extends Apache-Airflow with CWL v1.0 support.
Installation
pip3.6 install -U cwl-airflow-parser
Requirements
Package has been tested on Ubuntu 16.04.3 and Mac OS X Sierra/ High Sierra. Make sure your system satisfies the following criteria:
- python 3.6
- docker
- nodejs
Usage
from cwl_airflow_parser import CWLDAG, CWLJobDispatcher, CWLJobGatherer
from datetime import timedelta
def cwl_workflow(workflow_file):
dag = CWLDAG(default_args={
'owner': 'airflow',
'email': ['my@email.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 20,
'retry_exponential_backoff': True,
'retry_delay': timedelta(minutes=30),
'max_retry_delay': timedelta(minutes=60 * 4)
},
cwl_workflow=workflow_file)
dag.create()
dag.add(CWLJobDispatcher(dag=dag), to='top')
dag.add(CWLJobGatherer(dag=dag), to='bottom')
return dag
cwl_workflow("/path/to/my/workflow.cwl")
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for cwl-airflow-parser-1.0.20181210183459.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 50be66d45e93b4357e7820329153396a8c8db896d88cb7e7d25fda73f66af958 |
|
MD5 | 4d5cdb6c3ec01f57a2d65daa999a5e49 |
|
BLAKE2b-256 | bdb590fd5c5ba8834861d03ee157877e34cd67e3e6c1d034d4d13edf2ee7a106 |