Back-ported airflow.providers.apache.spark.* package for Airflow 1.10.*
Project description
Package apache-airflow-backport-providers-apache-spark
Release: 2020.5.20
Backport package
This is a backport providers package for apache.spark
provider. All classes for this provider package
are in airflow.providers.apache.spark
python package.
Only Python 3.6+ is supported for this backport package.
While Airflow 1.10.* continues to support Python 2.7+ - you need to upgrade python to 3.6+ if you want to use this backport package.
Installation
You can install this package on top of an existing airflow 1.10.* installation via
pip install apache-airflow-backport-providers-apache-spark
Compatibility
For full compatibility and test status of the backport packages check Airflow Backport Package Compatibility
PIP requirements
PIP package | Version required |
---|---|
pyspark |
Provider class summary
All classes in Airflow 2.0 are in airflow.providers.apache.spark
package.
Operators
Moved operators
Airflow 2.0 operators: airflow.providers.apache.spark package |
Airflow 1.10.* previous location (usually airflow.contrib ) |
---|---|
operators.spark_jdbc.SparkJDBCOperator | contrib.operators.spark_jdbc_operator.SparkJDBCOperator |
operators.spark_sql.SparkSqlOperator | contrib.operators.spark_sql_operator.SparkSqlOperator |
operators.spark_submit.SparkSubmitOperator | contrib.operators.spark_submit_operator.SparkSubmitOperator |
Hooks
Moved hooks
Airflow 2.0 hooks: airflow.providers.apache.spark package |
Airflow 1.10.* previous location (usually airflow.contrib ) |
---|---|
hooks.spark_jdbc.SparkJDBCHook | contrib.hooks.spark_jdbc_hook.SparkJDBCHook |
hooks.spark_sql.SparkSqlHook | contrib.hooks.spark_sql_hook.SparkSqlHook |
hooks.spark_submit.SparkSubmitHook | contrib.hooks.spark_submit_hook.SparkSubmitHook |
Releases
Release 2020.5.20
Commit | Committed | Subject |
---|---|---|
00642a46d | 2020-05-26 | Fixed name of 20 remaining wrongly named operators. (#8994) |
375d1ca22 | 2020-05-19 | Release candidate 2 for backport packages 2020.05.20 (#8898) |
12c5e5d8a | 2020-05-17 | Prepare release candidate for backport packages (#8891) |
f3521fb0e | 2020-05-16 | Regenerate readme files for backport package release (#8886) |
92585ca4c | 2020-05-15 | Added automated release notes generation for backport operators (#8807) |
7506c73f1 | 2020-05-10 | Add default conf parameter to Spark JDBC Hook (#8787) |
487b5cc50 | 2020-05-06 | Add guide for Apache Spark operators (#8305) |
87969a350 | 2020-04-09 | [AIRFLOW-6515] Change Log Levels from Info/Warn to Error (#8170) |
be1451b0e | 2020-04-02 | [AIRFLOW-7026] Improve SparkSqlHook's error message (#7749) |
4bde99f13 | 2020-03-23 | Make airflow/providers pylint compatible (#7802) |
7e6372a68 | 2020-03-23 | Add call to Super call in apache providers (#7820) |
2327aa5a2 | 2020-03-12 | [AIRFLOW-7025] Fix SparkSqlHook.run_query to handle its parameter properly (#7677) |
024b4bf96 | 2020-03-10 | [AIRFLOW-7024] Add the verbose parameter support to SparkSqlOperator (#7676) |
b59042b5a | 2020-02-28 | [AIRFLOW-6949] Respect explicit spark.kubernetes.namespace conf to SparkSubmitOperator (#7575) |
97a429f9d | 2020-02-02 | [AIRFLOW-6714] Remove magic comments about UTF-8 (#7338) |
0481b9a95 | 2020-01-12 | [AIRFLOW-6539][AIP-21] Move Apache classes to providers.apache package (#7142) |
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for apache-airflow-backport-providers-apache-spark-2020.5.20rc3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0a3f7acea11106da26fbd619922e60f6f1d58316c7cef3bccf9405a8899b377 |
|
MD5 | c4ffc24e8fde45c02a0be33d75679ddb |
|
BLAKE2b-256 | 1e028c47437b5b516045120a46ce05939894c0838676d1769cdaa0567572745f |
Hashes for apache_airflow_backport_providers_apache_spark-2020.5.20rc3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68dcb491049e56af5a824800f14cb8745dd0428a941689cb3c44737a15b85f2a |
|
MD5 | b724653c59aade32db2a50bd72cdb5ea |
|
BLAKE2b-256 | e0dac5183969260c4403cd764e9adb272d668476ea617efe48f1beefeb95601e |