Data Mechanics Airflow Plugin

An Airflow plugin to launch and monitor Spark applications on the Data Mechanics platform.

Environment

• Python >= 3.5
• apache-airflow >= 1.10.x. Compatible with Airflow 2.

Installation and usage

A tutorial to configure and use this plugin is available in the Data Mechanics docs.

The main difference between Airflow 1 and Airflow 2 is how to import the plugin:

# Airflow 1
from airflow.operators.datamechanics import DataMechanicsOperator

# Airflow 2
from datamechanics_airflow_plugin.operator import DataMechanicsOperator

Example DAGs

You can see example DAGs for Airflow 1 and Airflow 2.

Development

Development instructions.

Changelog

1.1.1 2021-03-26

Fixed

• Kill an app instead of deleting it when the Airflow task is marked as failed

1.1.0 2021-03-10

Added

• Support for Airflow templating in Data Mechanics operator's arguments (by @jj-ookla)
• Minimal support for Airflow 2 (the code is unchanged, only the doc and the contributor dev environment have changed)

1.0.7 2020-09-11

Changed

• Changed bumversion config

1.0.6 2020-09-11

Changed

• Updated doc

1.0.5 2020-09-11

Fixed

• Dependency management

1.0.0 2020-09-11

Changed

• Converted the existing plugin into a Python package