Package extends Airflow functionality with CWL v1.0 support

Project description

cwl-airflow-parser

About

Extends Apache-Airflow with CWL v1.0 support.

Installation

pip3.6 install -U cwl-airflow-parser

Requirements

Package has been tested on Ubuntu 16.04.3 and Mac OS X Sierra/ High Sierra. Make sure your system satisfies the following criteria:

python 3.6
docker
nodejs

Usage

from cwl_airflow_parser import CWLDAG, CWLJobDispatcher, CWLJobGatherer
from datetime import timedelta

def cwl_workflow(workflow_file):
    dag = CWLDAG(default_args={
        'owner': 'airflow',
        'email': ['my@email.com'],
        'email_on_failure': False,
        'email_on_retry': False,
        'retries': 20,
        'retry_exponential_backoff': True,
        'retry_delay': timedelta(minutes=30),
        'max_retry_delay': timedelta(minutes=60 * 4)
    },
        cwl_workflow=workflow_file)
    dag.create()
    dag.add(CWLJobDispatcher(dag=dag), to='top')
    dag.add(CWLJobGatherer(dag=dag), to='bottom')

    return dag
 
cwl_workflow("/path/to/my/workflow.cwl")

Project details

Release history Release notifications

This version

1.0.20181213213631

Dec 13, 2018

1.0.20181210183459

Dec 11, 2018

1.0.20180918220755

Sep 18, 2018

1.0.20180622010727

Jun 22, 2018

1.0.20180220071453

Feb 20, 2018

1.0.20180215072308

Feb 15, 2018

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for cwl-airflow-parser, version 1.0.20181213213631
Filename, size	File type	Python version	Upload date	Hashes
Filename, size cwl-airflow-parser-1.0.20181213213631.tar.gz (13.7 kB)	File type Source	Python version None	Upload date Dec 13, 2018	Hashes View hashes

Hashes for cwl-airflow-parser-1.0.20181213213631.tar.gz

Hashes for cwl-airflow-parser-1.0.20181213213631.tar.gz
Algorithm	Hash digest
SHA256	`b4c3e6ac4cd86dbe2f0d7fda4c608e580f018c3608f01611713daff1250ebe25`
MD5	`f651352ecad22e94dfb135654c96fcfe`
BLAKE2-256	`06f79baaec92b2a85930e12c140e24acd41a6dd751609b07e95b0240c3d21404`