Package extends Airflow functionality with CWL v1.0 support
Project description
cwl-airflow-parser
About
Extends Apache-Airflow with CWL v1.0 support.
Installation
pip3.6 install -U cwl-airflow-parser
Requirements
Package has been tested on Ubuntu 16.04.3 and Mac OS X Sierra/ High Sierra. Make sure your system satisfies the following criteria:
- python 3.6
- docker
- nodejs
Usage
from cwl_airflow_parser import CWLDAG, CWLJobDispatcher, CWLJobGatherer from datetime import timedelta def cwl_workflow(workflow_file): dag = CWLDAG(default_args={ 'owner': 'airflow', 'email': ['my@email.com'], 'email_on_failure': False, 'email_on_retry': False, 'retries': 20, 'retry_exponential_backoff': True, 'retry_delay': timedelta(minutes=30), 'max_retry_delay': timedelta(minutes=60 * 4) }, cwl_workflow=workflow_file) dag.create() dag.add(CWLJobDispatcher(dag=dag), to='top') dag.add(CWLJobGatherer(dag=dag), to='bottom') return dag cwl_workflow("/path/to/my/workflow.cwl")
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size cwl-airflow-parser-1.0.20181213213631.tar.gz (13.7 kB) | File type Source | Python version None | Upload date | Hashes View |
Close
Hashes for cwl-airflow-parser-1.0.20181213213631.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4c3e6ac4cd86dbe2f0d7fda4c608e580f018c3608f01611713daff1250ebe25 |
|
MD5 | f651352ecad22e94dfb135654c96fcfe |
|
BLAKE2-256 | 06f79baaec92b2a85930e12c140e24acd41a6dd751609b07e95b0240c3d21404 |