datamechanics-airflow-plugin 1.1.1

An Airflow plugin to launch and monitor Spark applications on the Data Mechanics platform

Data Mechanics Airflow Plugin

An Airflow plugin to launch and monitor Spark applications on the Data Mechanics platform.

Environment

• Python >= 3.5
• apache-airflow >= 1.10.x. Compatible with Airflow 2.

Installation and usage

A tutorial to configure and use this plugin is available in the Data Mechanics docs.

The main difference between Airflow 1 and Airflow 2 is how to import the plugin:

# Airflow 1
from airflow.operators.datamechanics import DataMechanicsOperator

# Airflow 2
from datamechanics_airflow_plugin.operator import DataMechanicsOperator

Example DAGs

You can see example DAGs for Airflow 1 and Airflow 2.

Development

Development instructions.

Changelog

1.1.1 2021-03-26

Fixed

• Kill an app instead of deleting it when the Airflow task is marked as failed

1.1.0 2021-03-10

Added

• Support for Airflow templating in Data Mechanics operator's arguments (by @jj-ookla)
• Minimal support for Airflow 2 (the code is unchanged, only the doc and the contributor dev environment have changed)

1.0.7 2020-09-11

Changed

• Changed bumversion config

1.0.6 2020-09-11

Changed

• Updated doc

1.0.5 2020-09-11

Fixed

• Dependency management

1.0.0 2020-09-11

Changed

• Converted the existing plugin into a Python package