Package extends Airflow functionality with CWL v1.0 support
Project description
# cwl-airflow-parser
### About
Extends **[Apache-Airflow](https://github.com/apache/incubator-airflow)** with **[CWL v1.0](http://www.commonwl.org/v1.0/)** support.
### Installation
```sh
pip3.6 install -U cwl-airflow-parser
```
### Requirements
Package has been tested on Ubuntu 16.04.3 and Mac OS X Sierra/ High Sierra. Make sure your system satisfies the following criteria:
- python 3.6
- [docker](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/)
- nodejs
### Usage
```python
from cwl_airflow_parser import CWLDAG, CWLJobDispatcher, CWLJobGatherer
from datetime import timedelta
def cwl_workflow(workflow_file):
dag = CWLDAG(default_args={
'owner': 'airflow',
'email': ['my@email.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 20,
'retry_exponential_backoff': True,
'retry_delay': timedelta(minutes=30),
'max_retry_delay': timedelta(minutes=60 * 4)
},
cwl_workflow=workflow_file)
dag.create()
dag.add(CWLJobDispatcher(dag=dag), to='top')
dag.add(CWLJobGatherer(dag=dag), to='bottom')
return dag
cwl_workflow("/path/to/my/workflow.cwl")
```
### About
Extends **[Apache-Airflow](https://github.com/apache/incubator-airflow)** with **[CWL v1.0](http://www.commonwl.org/v1.0/)** support.
### Installation
```sh
pip3.6 install -U cwl-airflow-parser
```
### Requirements
Package has been tested on Ubuntu 16.04.3 and Mac OS X Sierra/ High Sierra. Make sure your system satisfies the following criteria:
- python 3.6
- [docker](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/)
- nodejs
### Usage
```python
from cwl_airflow_parser import CWLDAG, CWLJobDispatcher, CWLJobGatherer
from datetime import timedelta
def cwl_workflow(workflow_file):
dag = CWLDAG(default_args={
'owner': 'airflow',
'email': ['my@email.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 20,
'retry_exponential_backoff': True,
'retry_delay': timedelta(minutes=30),
'max_retry_delay': timedelta(minutes=60 * 4)
},
cwl_workflow=workflow_file)
dag.create()
dag.add(CWLJobDispatcher(dag=dag), to='top')
dag.add(CWLJobGatherer(dag=dag), to='bottom')
return dag
cwl_workflow("/path/to/my/workflow.cwl")
```
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file cwl-airflow-parser-1.0.20180622010727.tar.gz
.
File metadata
- Download URL: cwl-airflow-parser-1.0.20180622010727.tar.gz
- Upload date:
- Size: 9.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f08233750e34fb5bbb2ef9203314ad0221c26117cd27071bb93859f97dab9d7 |
|
MD5 | 899ae649006a6890dbbb25254104bf99 |
|
BLAKE2b-256 | 9500d34bb69b5de0c4ffbbbc9f22007c3cd0c99619b7d2856e7a34524c71712e |