Skip to main content

Provider for Apache Airflow. Implements apache-airflow-providers-apache-spark package

Project description

Package apache-airflow-providers-apache-spark

Release: 2.1.3

Apache Spark

Provider package

This is a provider package for apache.spark provider. All classes for this provider package are in airflow.providers.apache.spark python package.

You can find package information and changelog for the provider in the documentation.

Installation

You can install this package on top of an existing Airflow 2.1+ installation via pip install apache-airflow-providers-apache-spark

The package supports the following python versions: 3.7,3.8,3.9,3.10

PIP requirements

PIP package Version required
apache-airflow >=2.1.0
pyspark  

Changelog

2.1.3

Bug Fixes

  • Fix mistakenly added install_requires for all providers (#22382)

2.1.2

Misc

  • Add Trove classifiers in PyPI (Framework :: Apache Airflow :: Provider)

2.1.1

Bug Fixes

  • fix param rendering in docs of SparkSubmitHook (#21788)

Misc

  • Support for Python 3.10

2.1.0

Features

  • Add more SQL template fields renderers (#21237)
  • Add optional features in providers. (#21074)

2.0.3

Bug Fixes

  • Ensure Spark driver response is valid before setting UNKNOWN status (#19978)

2.0.2

Bug Fixes

  • fix bug of SparkSql Operator log  going to infinite loop. (#19449)

2.0.1

Misc

  • Optimise connection importing for Airflow 2.2.0

2.0.0

Breaking changes

  • Auto-apply apply_default decorator (#15667)

Warning

Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+. If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded automatically and you will have to manually run airflow upgrade db to complete the migration.

Bug fixes

  • Make SparkSqlHook use Connection (#15794)

1.0.3

Bug fixes

  • Fix 'logging.exception' redundancy (#14823)

1.0.2

Bug fixes

  • Use apache.spark provider without kubernetes (#14187)

1.0.1

Updated documentation and readme files.

1.0.0

Initial version of the provider.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Built Distribution

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page