Skip to main content

Package extends Airflow functionality with CWL v1.0 support

Project description

cwl-airflow-parser

About

Extends Apache-Airflow with CWL v1.0 support.

Installation

pip3.6 install -U cwl-airflow-parser

Requirements

Package has been tested on Ubuntu 16.04.3 and Mac OS X Sierra/ High Sierra. Make sure your system satisfies the following criteria:

Usage

from cwl_airflow_parser import CWLDAG, CWLJobDispatcher, CWLJobGatherer
from datetime import timedelta

def cwl_workflow(workflow_file):
    dag = CWLDAG(default_args={
        'owner': 'airflow',
        'email': ['my@email.com'],
        'email_on_failure': False,
        'email_on_retry': False,
        'retries': 20,
        'retry_exponential_backoff': True,
        'retry_delay': timedelta(minutes=30),
        'max_retry_delay': timedelta(minutes=60 * 4)
    },
        cwl_workflow=workflow_file)
    dag.create()
    dag.add(CWLJobDispatcher(dag=dag), to='top')
    dag.add(CWLJobGatherer(dag=dag), to='bottom')

    return dag
 
cwl_workflow("/path/to/my/workflow.cwl")

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
cwl-airflow-parser-1.0.20180918220755.tar.gz (9.6 kB) Copy SHA256 hash SHA256 Source None Sep 18, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page