Skip to main content

An Airflow plugin to launch and monitor Spark applications on the Data Mechanics platform

Project description

Data Mechanics Airflow Plugin

An Airflow plugin to launch and monitor Spark applications on the Data Mechanics platform.

Environment

  • Python >= 3.5
  • apache-airflow >= 1.10.x. Compatible with Airflow 2.

Installation and usage

A tutorial to configure and use this plugin is available in the Data Mechanics docs.

The main difference between Airflow 1 and Airflow 2 is how to import the plugin:

# Airflow 1
from airflow.operators.datamechanics import DataMechanicsOperator

# Airflow 2
from datamechanics_airflow_plugin.operator import DataMechanicsOperator

Example DAGs

You can see example DAGs for Airflow 1 and Airflow 2.

Development

Development instructions.

Changelog

1.1.1 2021-03-26

Fixed

  • Kill an app instead of deleting it when the Airflow task is marked as failed

1.1.0 2021-03-10

Added

  • Support for Airflow templating in Data Mechanics operator's arguments (by @jj-ookla)
  • Minimal support for Airflow 2 (the code is unchanged, only the doc and the contributor dev environment have changed)

1.0.7 2020-09-11

Changed

  • Changed bumversion config

1.0.6 2020-09-11

Changed

  • Updated doc

1.0.5 2020-09-11

Fixed

  • Dependency management

1.0.0 2020-09-11

Changed

  • Converted the existing plugin into a Python package

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datamechanics_airflow_plugin-1.1.1.tar.gz (6.4 kB view hashes)

Uploaded source

Built Distribution

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page