airflow-hdinsight
A set of airflow hooks, operators and sensors to allow airflow DAGs to operate with the Azure HDInsight platform, for cluster creation and monitoring as well as job submission and monitoring. Also included are some enhanced Azure Blob and Data Lake sensors.
This project is both an amalgamation and enhancement of existing open source airflow extensions, plus new extensions to solve the problem.
Installation
pip install airflow-hdinsight
Extensions
airflowhdi
| Type | Name | What it does |
|---|---|---|
| Hook | AzureHDInsightHook | Uses the HDInsightManagementClient from the HDInsight SDK for Python to expose several operations on an HDInsight cluster - get cluster state, create, delete. |
| Operator | AzureHDInsightCreateClusterOperator | Use the AzureHDInsightHook to create a cluster |
| Operator | AzureHDInsightDeleteClusterOperator | Use the AzureHDInsightHook to delete a cluster |
| Operator | ConnectedAzureHDInsightCreateClusterOperator | Extends the AzureHDInsightCreateClusterOperator to allow fetching of the security credentials and cluster creation spec from an airflow connection |
| Operator | AzureHDInsightSshOperator | Uses the AzureHDInsightHook and SSHHook to run an SSH command on the master node of the given HDInsight cluster |
| Sensor | AzureHDInsightClusterSensor | A sensor to monitor the provisioning state or running state (can switch between either mode) of a given HDInsight cluster. Uses the AzureHDInsightHook. |
| Sensor | WasbWildcardPrefixSensor | An enhancement to the WasbPrefixSensor to support sensing on a wildcard prefix |
| Sensor | AzureDataLakeStorageGen1WebHdfsSensor | Uses airflow's AzureDataLakeHook to sense a glob path (which implicitly supports wildcards) on ADLS Gen 1. ADLS Gen 2 is not yet supported in airflow. |
airflowlivy
| Type | Name | What it does |
|---|---|---|
| Hook | LivyBatchHook | Uses the Apache Livy Batch API to submit spark jobs to a livy server, get batch state, verify batch state by quering either the spark history server or yarn resource manager, spill the logs of the spark job post completion, etc. |
| Operator | LivyBatchOperator | Uses the LivyBatchHook to submit a spark job to a livy server |
| Sensor | LivyBatchSensor | Uses the LivyBatchHook to sense termination and verify completion, spill logs of a spark job submitted earlier to a livy server |
Origins of the HDinsight operator work
The HDInsight operator work is loosely inspired from alikemalocalan/airflow-hdinsight-operators, however that has a huge number of defects, as to why it was never accepted to be merged into airflow in the first place. This project solves all of those issues and more, and is frankly a full rewrite.
Origins of the livy work
The livy batch operator is based on the work by panovvv's project airfllow-livy-operators. It does some necessary changes:
- Seperates the operator into a hook (LivyBatchHook), an operator (LivyBatchOperator) and a sensor (LivyBatchSensor)
- Adds additional verification and log spilling to the sensor (the original sensor does not)
- Removes additional verifiation and log spilling from the operator - hence alllowing a async pattern akin to the EMR add step operator and step sensor.
- Creates livy, spark and YARN airflow connections dynamically from an Azure HDInsight connection
- Returns the batch ID from the operator so that a sensor can use it after being passed through XCom
- Changes logging to LoggingMixin calls
- Allows templatization of fields
State of airflow livy operators in the wild..
As it stands today (June of 2020), there are multiple airflow livy operator projects out there:
- panovvv/airflow-livy-operators: the project which this project bases its work on
- the official livy provider in airflow 2.0, with a backport available for airflow 1.1.x: alas the official provider has very limited functionality - it does not spill the job's logs, and it does not do additional verification for job completion using spark history server or yarn resource manager, amongst other limitations
- rssanders3/airflow-spark-operator-plugin: this is the oldest livy operator, which only supports livy sessions and not batches. there's a copy of this in alikemalocalan/airflow-hdinsight-operators.
1 maintainer
Metadata
1 maintainer
Release files for airflow-hdinsight 0.0.1.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| airflow_hdinsight-0.0.1.3-py3-none-any.whl | Python 3 | none | any | Details for airflow_hdinsight-0.0.1.3-py3-none-any.whl |
Release files / airflow_hdinsight-0.0.1.3-py3-none-any.whl
| Download URL | airflow_hdinsight-0.0.1.3-py3-none-any.whl |
|---|---|
| Size | 31.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c2184eb1cbfde1ea9bd16d66cfe67045dcfdec3b7e03973e1fd6c7fe5cab2fde
|
|
BLAKE2b-256 checksum How to use checksums |
cf6bd8075b98de5c244c314539bfe50ac0d37eabed753ec48207a84380fd6471
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.1 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
Security for airflow-hdinsight 0.0.1.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
