airflow-hdinsight 0.0.1.3

HDInsight provider for Airflow

airflow-hdinsight

A set of airflow hooks, operators and sensors to allow airflow DAGs to operate with the Azure HDInsight platform, for cluster creation and monitoring as well as job submission and monitoring. Also included are some enhanced Azure Blob and Data Lake sensors.

This project is both an amalgamation and enhancement of existing open source airflow extensions, plus new extensions to solve the problem.

Installation

pip install airflow-hdinsight

Extensions

airflowhdi

Type	Name	What it does
Hook	AzureHDInsightHook	Uses the HDInsightManagementClient from the HDInsight SDK for Python to expose several operations on an HDInsight cluster - get cluster state, create, delete.
Operator	AzureHDInsightCreateClusterOperator	Use the AzureHDInsightHook to create a cluster
Operator	AzureHDInsightDeleteClusterOperator	Use the AzureHDInsightHook to delete a cluster
Operator	ConnectedAzureHDInsightCreateClusterOperator	Extends the AzureHDInsightCreateClusterOperator to allow fetching of the security credentials and cluster creation spec from an airflow connection
Operator	AzureHDInsightSshOperator	Uses the AzureHDInsightHook and SSHHook to run an SSH command on the master node of the given HDInsight cluster
Sensor	AzureHDInsightClusterSensor	A sensor to monitor the provisioning state or running state (can switch between either mode) of a given HDInsight cluster. Uses the AzureHDInsightHook.
Sensor	WasbWildcardPrefixSensor	An enhancement to the WasbPrefixSensor to support sensing on a wildcard prefix
Sensor	AzureDataLakeStorageGen1WebHdfsSensor	Uses airflow's AzureDataLakeHook to sense a glob path (which implicitly supports wildcards) on ADLS Gen 1. ADLS Gen 2 is not yet supported in airflow.

airflowlivy

Type	Name	What it does
Hook	LivyBatchHook	Uses the Apache Livy Batch API to submit spark jobs to a livy server, get batch state, verify batch state by quering either the spark history server or yarn resource manager, spill the logs of the spark job post completion, etc.
Operator	LivyBatchOperator	Uses the LivyBatchHook to submit a spark job to a livy server
Sensor	LivyBatchSensor	Uses the LivyBatchHook to sense termination and verify completion, spill logs of a spark job submitted earlier to a livy server

Origins of the HDinsight operator work

The HDInsight operator work is loosely inspired from alikemalocalan/airflow-hdinsight-operators, however that has a huge number of defects, as to why it was never accepted to be merged into airflow in the first place. This project solves all of those issues and more, and is frankly a full rewrite.

Origins of the livy work

The livy batch operator is based on the work by panovvv's project airfllow-livy-operators. It does some necessary changes:

• Seperates the operator into a hook (LivyBatchHook), an operator (LivyBatchOperator) and a sensor (LivyBatchSensor)
• Adds additional verification and log spilling to the sensor (the original sensor does not)
• Removes additional verifiation and log spilling from the operator - hence alllowing a async pattern akin to the EMR add step operator and step sensor.
• Creates livy, spark and YARN airflow connections dynamically from an Azure HDInsight connection
• Returns the batch ID from the operator so that a sensor can use it after being passed through XCom
• Changes logging to LoggingMixin calls
• Allows templatization of fields

State of airflow livy operators in the wild..

As it stands today (June of 2020), there are multiple airflow livy operator projects out there:

• panovvv/airflow-livy-operators: the project which this project bases its work on
• the official livy provider in airflow 2.0, with a backport available for airflow 1.1.x: alas the official provider has very limited functionality - it does not spill the job's logs, and it does not do additional verification for job completion using spark history server or yarn resource manager, amongst other limitations
• rssanders3/airflow-spark-operator-plugin: this is the oldest livy operator, which only supports livy sessions and not batches. there's a copy of this in alikemalocalan/airflow-hdinsight-operators.