apache-airflow-providers-apache-spark 2.1.1

Provider package apache-airflow-providers-apache-spark for Apache Airflow

Package apache-airflow-providers-apache-spark

Release: 2.1.1

Apache Spark

Provider package

This is a provider package for apache.spark provider. All classes for this provider package are in airflow.providers.apache.spark python package.

You can find package information and changelog for the provider in the documentation.

Installation

You can install this package on top of an existing Airflow 2.1+ installation via pip install apache-airflow-providers-apache-spark

The package supports the following python versions: 3.7,3.8,3.9,3.10

PIP requirements

PIP package	Version required
apache-airflow	>=2.1.0
pyspark

Changelog

2.1.1

Bug Fixes

• fix param rendering in docs of SparkSubmitHook (#21788)

Misc

• Support for Python 3.10

2.1.0

Features

• Add more SQL template fields renderers (#21237)

• Add optional features in providers. (#21074)

2.0.3

Bug Fixes

• Ensure Spark driver response is valid before setting UNKNOWN status (#19978)

2.0.2

Bug Fixes

• fix bug of SparkSql Operator log going to infinite loop. (#19449)

2.0.1

Misc

• Optimise connection importing for Airflow 2.2.0

2.0.0

Breaking changes

• Auto-apply apply_default decorator (#15667)

Warning

Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+. If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded automatically and you will have to manually run airflow upgrade db to complete the migration.

Bug fixes

• Make SparkSqlHook use Connection (#15794)

1.0.3

Bug fixes

• Fix 'logging.exception' redundancy (#14823)

1.0.2

Bug fixes

• Use apache.spark provider without kubernetes (#14187)

1.0.1

Updated documentation and readme files.

1.0.0

Initial version of the provider.