An Airflow plugin to launch and monitor Spark applications on the Data Mechanics platform
Project description
Data Mechanics Airflow Plugin
An Airflow plugin to launch and monitor Spark applications on the Data Mechanics platform.
Environment
- Python >= 3.5
- apache-airflow >= 1.10.x. Compatible with Airflow 2.
Installation and usage
A tutorial to configure and use this plugin is available in the Data Mechanics docs.
The main difference between Airflow 1 and Airflow 2 is how to import the plugin:
# Airflow 1
from airflow.operators.datamechanics import DataMechanicsOperator
# Airflow 2
from datamechanics_airflow_plugin.operator import DataMechanicsOperator
Example DAGs
You can see example DAGs for Airflow 1 and Airflow 2.
Development
Changelog
1.1.1 2021-03-26
Fixed
- Kill an app instead of deleting it when the Airflow task is marked as failed
1.1.0 2021-03-10
Added
- Support for Airflow templating in Data Mechanics operator's arguments (by @jj-ookla)
- Minimal support for Airflow 2 (the code is unchanged, only the doc and the contributor dev environment have changed)
1.0.7 2020-09-11
Changed
- Changed bumversion config
1.0.6 2020-09-11
Changed
- Updated doc
1.0.5 2020-09-11
Fixed
- Dependency management
1.0.0 2020-09-11
Changed
- Converted the existing plugin into a Python package
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for datamechanics_airflow_plugin-1.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | dab3f764eeaf2ba2fca014932d5710c480cbf7a8931de6bb6eca3f936f9f7533 |
|
MD5 | a7b749696b48db2e68cf05850a9cdd00 |
|
BLAKE2b-256 | e276ef56f66e13521d7099eaf825031eb5f4d71f6a0bdbcede1cfeb744ad368d |
Close
Hashes for datamechanics_airflow_plugin-1.1.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7793997ec6f0e43535bbe53c6e73ab0b93cda0bab2937ac664594ace96a2786e |
|
MD5 | d0f4f399fd8191ed83f3d42f24d27ca6 |
|
BLAKE2b-256 | 14132d21592e7908fc444617bf9eec12ff29cbdc93d8647d3d0a457a2ac53129 |