Skip to main content

Backport provider package apache-airflow-backport-providers-amazon for Apache Airflow

Project description

Package apache-airflow-backport-providers-amazon

Release: 2021.3.3

Table of contents

Backport package

This is a backport providers package for amazon provider. All classes for this provider package are in airflow.providers.amazon python package.

Only Python 3.6+ is supported for this backport package.

While Airflow 1.10.* continues to support Python 2.7+ - you need to upgrade python to 3.6+ if you want to use this backport package.

Release 2021.3.3

Features

  • Adding support to put extra arguments for Glue Job. (#14027)
  • Avoid using threads in S3 remote logging upload (#14414)
  • Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query (#14177)
  • includes the STS token if STS credentials are used (#11227)

Release 2021.2.5

Features

  • Add aws ses email backend for use with EmailOperator. (#13986)
  • Add bucket_name to template fileds in S3 operators (#13973)
  • Add ExasolToS3Operator (#13847)
  • AWS Glue Crawler Integration (#13072)
  • Add acl_policy to S3CopyObjectOperator (#13773)
  • AllowDiskUse parameter and docs in MongotoS3Operator (#12033)
  • Add S3ToFTPOperator (#11747)
  • add xcom push for ECSOperator (#12096)
  • [AIRFLOW-3723] Add Gzip capability to mongo_to_S3 operator (#13187)
  • Add S3KeySizeSensor (#13049)
  • Add 'mongo_collection' to template_fields in MongoToS3Operator (#13361)
  • Allow Tags on AWS Batch Job Submission (#13396)

Bug fixes

  • Fix bug in GCSToS3Operator (#13718)
  • Fix S3KeysUnchangedSensor so that template_fields work (#13490)

Change in import paths

If you are upgrading from 2020.10.5 note the following changes in import paths

Old path New path
airflow.providers.amazon.aws.hooks.aws_dynamodb.AwsDynamoDBHook airflow.providers.amazon.aws.hooks.dynamodb.AwsDynamoDBHook

Installation

You can install this package on top of an existing airflow 1.10.* installation via pip install apache-airflow-backport-providers-amazon

PIP requirements

PIP package Version required
boto3 >=1.15.0,<1.16.0
botocore >=1.18.0,<1.19.0
watchtower ~=0.7.3

Cross provider package dependencies

Those are dependencies that might be needed in order to use all the features of the package. You need to install the specified backport providers package in order to use them.

You can install such cross-provider dependencies when installing from PyPI. For example:

pip install apache-airflow-backport-providers-amazon[apache.hive]
Dependent package Extra
apache-airflow-backport-providers-apache-hive apache.hive
apache-airflow-backport-providers-exasol exasol
apache-airflow-backport-providers-ftp ftp
apache-airflow-backport-providers-google google
apache-airflow-backport-providers-imap imap
apache-airflow-backport-providers-mongo mongo
apache-airflow-backport-providers-mysql mysql
apache-airflow-backport-providers-postgres postgres
apache-airflow-backport-providers-ssh ssh

Provider classes summary

In Airflow 2.0, all operators, transfers, hooks, sensors, secrets for the amazon provider are in the airflow.providers.amazon package. You can read more about the naming conventions used in Naming conventions for provider packages

Operators

New operators

New Airflow 2.0 operators: airflow.providers.amazon package
aws.operators.cloud_formation.CloudFormationCreateStackOperator
aws.operators.cloud_formation.CloudFormationDeleteStackOperator
aws.operators.datasync.AWSDataSyncOperator
aws.operators.ec2_start_instance.EC2StartInstanceOperator
aws.operators.ec2_stop_instance.EC2StopInstanceOperator
aws.operators.emr_modify_cluster.EmrModifyClusterOperator
aws.operators.glacier.GlacierCreateJobOperator
aws.operators.glue.AwsGlueJobOperator
aws.operators.glue_crawler.AwsGlueCrawlerOperator
aws.operators.s3_bucket.S3CreateBucketOperator
aws.operators.s3_bucket.S3DeleteBucketOperator
aws.operators.s3_file_transform.S3FileTransformOperator
aws.operators.sagemaker_processing.SageMakerProcessingOperator
aws.operators.step_function_get_execution_output.StepFunctionGetExecutionOutputOperator
aws.operators.step_function_start_execution.StepFunctionStartExecutionOperator

Moved operators

Airflow 2.0 operators: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.operators.athena.AWSAthenaOperator contrib.operators.aws_athena_operator.AWSAthenaOperator
aws.operators.batch.AwsBatchOperator contrib.operators.awsbatch_operator.AWSBatchOperator
aws.operators.ecs.ECSOperator contrib.operators.ecs_operator.ECSOperator
aws.operators.emr_add_steps.EmrAddStepsOperator contrib.operators.emr_add_steps_operator.EmrAddStepsOperator
aws.operators.emr_create_job_flow.EmrCreateJobFlowOperator contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator
aws.operators.emr_terminate_job_flow.EmrTerminateJobFlowOperator contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator
aws.operators.s3_copy_object.S3CopyObjectOperator contrib.operators.s3_copy_object_operator.S3CopyObjectOperator
aws.operators.s3_delete_objects.S3DeleteObjectsOperator contrib.operators.s3_delete_objects_operator.S3DeleteObjectsOperator
aws.operators.s3_list.S3ListOperator contrib.operators.s3_list_operator.S3ListOperator
aws.operators.sagemaker_base.SageMakerBaseOperator contrib.operators.sagemaker_base_operator.SageMakerBaseOperator
aws.operators.sagemaker_endpoint.SageMakerEndpointOperator contrib.operators.sagemaker_endpoint_operator.SageMakerEndpointOperator
aws.operators.sagemaker_endpoint_config.SageMakerEndpointConfigOperator contrib.operators.sagemaker_endpoint_config_operator.SageMakerEndpointConfigOperator
aws.operators.sagemaker_model.SageMakerModelOperator contrib.operators.sagemaker_model_operator.SageMakerModelOperator
aws.operators.sagemaker_training.SageMakerTrainingOperator contrib.operators.sagemaker_training_operator.SageMakerTrainingOperator
aws.operators.sagemaker_transform.SageMakerTransformOperator contrib.operators.sagemaker_transform_operator.SageMakerTransformOperator
aws.operators.sagemaker_tuning.SageMakerTuningOperator contrib.operators.sagemaker_tuning_operator.SageMakerTuningOperator
aws.operators.sns.SnsPublishOperator contrib.operators.sns_publish_operator.SnsPublishOperator
aws.operators.sqs.SQSPublishOperator contrib.operators.aws_sqs_publish_operator.SQSPublishOperator

Transfer operators

New transfer operators

New Airflow 2.0 transfers: airflow.providers.amazon package
aws.transfers.exasol_to_s3.ExasolToS3Operator
aws.transfers.glacier_to_gcs.GlacierToGCSOperator
aws.transfers.mysql_to_s3.MySQLToS3Operator
aws.transfers.s3_to_ftp.S3ToFTPOperator

Moved transfer operators

Airflow 2.0 transfers: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.transfers.dynamodb_to_s3.DynamoDBToS3Operator contrib.operators.dynamodb_to_s3.DynamoDBToS3Operator
aws.transfers.gcs_to_s3.GCSToS3Operator operators.gcs_to_s3.GCSToS3Operator
aws.transfers.google_api_to_s3.GoogleApiToS3Operator operators.google_api_to_s3_transfer.GoogleApiToS3Transfer
aws.transfers.hive_to_dynamodb.HiveToDynamoDBOperator contrib.operators.hive_to_dynamodb.HiveToDynamoDBOperator
aws.transfers.imap_attachment_to_s3.ImapAttachmentToS3Operator contrib.operators.imap_attachment_to_s3_operator.ImapAttachmentToS3Operator
aws.transfers.mongo_to_s3.MongoToS3Operator contrib.operators.mongo_to_s3.MongoToS3Operator
aws.transfers.redshift_to_s3.RedshiftToS3Operator operators.redshift_to_s3_operator.RedshiftToS3Transfer
aws.transfers.s3_to_redshift.S3ToRedshiftOperator operators.s3_to_redshift_operator.S3ToRedshiftTransfer
aws.transfers.s3_to_sftp.S3ToSFTPOperator contrib.operators.s3_to_sftp_operator.S3ToSFTPOperator
aws.transfers.sftp_to_s3.SFTPToS3Operator contrib.operators.sftp_to_s3_operator.SFTPToS3Operator

Sensors

New sensors

New Airflow 2.0 sensors: airflow.providers.amazon package
aws.sensors.cloud_formation.CloudFormationCreateStackSensor
aws.sensors.cloud_formation.CloudFormationDeleteStackSensor
aws.sensors.ec2_instance_state.EC2InstanceStateSensor
aws.sensors.glacier.GlacierJobOperationSensor
aws.sensors.glue.AwsGlueJobSensor
aws.sensors.glue_crawler.AwsGlueCrawlerSensor
aws.sensors.redshift.AwsRedshiftClusterSensor
aws.sensors.s3_key.S3KeySizeSensor
aws.sensors.s3_keys_unchanged.S3KeysUnchangedSensor
aws.sensors.sagemaker_training.SageMakerTrainingSensor
aws.sensors.step_function_execution.StepFunctionExecutionSensor

Moved sensors

Airflow 2.0 sensors: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.sensors.athena.AthenaSensor contrib.sensors.aws_athena_sensor.AthenaSensor
aws.sensors.emr_base.EmrBaseSensor contrib.sensors.emr_base_sensor.EmrBaseSensor
aws.sensors.emr_job_flow.EmrJobFlowSensor contrib.sensors.emr_job_flow_sensor.EmrJobFlowSensor
aws.sensors.emr_step.EmrStepSensor contrib.sensors.emr_step_sensor.EmrStepSensor
aws.sensors.glue_catalog_partition.AwsGlueCatalogPartitionSensor contrib.sensors.aws_glue_catalog_partition_sensor.AwsGlueCatalogPartitionSensor
aws.sensors.s3_key.S3KeySensor sensors.s3_key_sensor.S3KeySensor
aws.sensors.s3_prefix.S3PrefixSensor sensors.s3_prefix_sensor.S3PrefixSensor
aws.sensors.sagemaker_base.SageMakerBaseSensor contrib.sensors.sagemaker_base_sensor.SageMakerBaseSensor
aws.sensors.sagemaker_endpoint.SageMakerEndpointSensor contrib.sensors.sagemaker_endpoint_sensor.SageMakerEndpointSensor
aws.sensors.sagemaker_transform.SageMakerTransformSensor contrib.sensors.sagemaker_transform_sensor.SageMakerTransformSensor
aws.sensors.sagemaker_tuning.SageMakerTuningSensor contrib.sensors.sagemaker_tuning_sensor.SageMakerTuningSensor
aws.sensors.sqs.SQSSensor contrib.sensors.aws_sqs_sensor.SQSSensor

Hooks

New hooks

New Airflow 2.0 hooks: airflow.providers.amazon package
aws.hooks.batch_client.AwsBatchClientHook
aws.hooks.batch_waiters.AwsBatchWaitersHook
aws.hooks.cloud_formation.AWSCloudFormationHook
aws.hooks.ec2.EC2Hook
aws.hooks.elasticache_replication_group.ElastiCacheReplicationGroupHook
aws.hooks.glacier.GlacierHook
aws.hooks.glue.AwsGlueJobHook
aws.hooks.glue_crawler.AwsGlueCrawlerHook
aws.hooks.kinesis.AwsFirehoseHook
aws.hooks.redshift.RedshiftHook
aws.hooks.secrets_manager.SecretsManagerHook
aws.hooks.ses.SESHook
aws.hooks.step_function.StepFunctionHook

Moved hooks

Airflow 2.0 hooks: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.hooks.athena.AWSAthenaHook contrib.hooks.aws_athena_hook.AWSAthenaHook
aws.hooks.base_aws.AwsBaseHook contrib.hooks.aws_hook.AwsHook
aws.hooks.datasync.AWSDataSyncHook contrib.hooks.aws_datasync_hook.AWSDataSyncHook
aws.hooks.dynamodb.AwsDynamoDBHook contrib.hooks.aws_dynamodb_hook.AwsDynamoDBHook
aws.hooks.emr.EmrHook contrib.hooks.emr_hook.EmrHook
aws.hooks.glue_catalog.AwsGlueCatalogHook contrib.hooks.aws_glue_catalog_hook.AwsGlueCatalogHook
aws.hooks.lambda_function.AwsLambdaHook contrib.hooks.aws_lambda_hook.AwsLambdaHook
aws.hooks.logs.AwsLogsHook contrib.hooks.aws_logs_hook.AwsLogsHook
aws.hooks.s3.S3Hook hooks.S3_hook.S3Hook
aws.hooks.sagemaker.SageMakerHook contrib.hooks.sagemaker_hook.SageMakerHook
aws.hooks.sns.AwsSnsHook contrib.hooks.aws_sns_hook.AwsSnsHook
aws.hooks.sqs.SQSHook contrib.hooks.aws_sqs_hook.SQSHook

Secrets

Moved secrets

Airflow 2.0 secrets: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.secrets.secrets_manager.SecretsManagerBackend contrib.secrets.aws_secrets_manager.SecretsManagerBackend
aws.secrets.systems_manager.SystemsManagerParameterStoreBackend contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend

Releases

Release 2021.3.3

Commit Committed Subject
13854c32a 2021-02-27 Adding support to put extra arguments for Glue Job. (#14027)
0d6cae417 2021-02-24 Avoid using threads in S3 remote logging uplod (#14414)
ca35bd7f7 2021-02-21 By default PIP will install all packages in .local folder (#14125)
1b1472630 2021-02-11 Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query (#14177)
9034f277e 2021-02-10 Document configuration for email backend credentials. (#14006)
8c5594b02 2021-02-08 includes the STS token if STS credentials are used (#11227)
cddbf9c11 2021-02-06 Use MongoDB color for MongoToS3Operator (#14103)
10343ec29 2021-02-05 Corrections in docs and tools after releasing provider RCs (#14082)

Release 2021.2.5

Commit Committed Subject
88bdcfa0d 2021-02-04 Prepare to release a new wave of providers. (#14013)
ac2f72c98 2021-02-01 Implement provider versioning tools (#13767)
01049ddce 2021-01-31 Add aws ses email backend for use with EmailOperator. (#13986)
ecfdc60bb 2021-01-29 Add bucket_name to template fileds in S3 operators (#13973)
d0ab7f6d3 2021-01-25 Add ExasolToS3Operator (#13847)
6d55f329f 2021-01-25 AWS Glue Crawler Integration (#13072)
f473ca713 2021-01-24 Replace &#39;google_cloud_storage_conn_id&#39; by &#39;gcp_conn_id&#39; when using &#39;GCSHook&#39; (#13851)
a9ac2b040 2021-01-23 Switch to f-strings using flynt. (#13732)
3fd5ef355 2021-01-21 Add missing logos for integrations (#13717)
29730d720 2021-01-20 Add acl_policy to S3CopyObjectOperator (#13773)
c065d3218 2021-01-19 AllowDiskUse parameter and docs in MongotoS3Operator (#12033)
ab5fe56ac 2021-01-16 Fix bug in GCSToS3Operator (#13718)
04d278f93 2021-01-13 Add S3ToFTPOperator (#11747)
8d42d9ed6 2021-01-11 add xcom push for ECSOperator (#12096)
308f1d066 2021-01-07 [AIRFLOW-3723] Add Gzip capability to mongo_to_S3 operator (#13187)
f69405fb0 2021-01-07 Fix S3KeysUnchangedSensor so that template_fields work (#13490)
4e479e1e1 2021-01-06 Add S3KeySizeSensor (#13049)
f7a1334ab 2021-01-02 Add &#39;mongo_collection&#39; to template_fields in MongoToS3Operator (#13361)
bd74eb0ca 2020-12-31 Allow Tags on AWS Batch Job Submission (#13396)
295d66f91 2020-12-30 Fix Grammar in PIP warning (#13380)
625576a3a 2020-12-18 Fix spelling (#13135)
6cf76d7ac 2020-12-18 Fix typo in pip upgrade command :( (#13148)
5090fb0c8 2020-12-15 Add script to generate integrations.json (#13073)
32971a1a2 2020-12-09 Updates providers versions to 1.0.0 (#12955)
d5589673a 2020-12-08 Move dummy_operator.py to dummy.py (#11178) (#11293)
b40dffa08 2020-12-08 Rename remaing modules to match AIP-21 (#12917)
9b39f2478 2020-12-08 Add support for dynamic connection form fields per provider (#12558)
bd90136aa 2020-11-30 Move operator guides to provider documentation packages (#12681)
02d94349b 2020-11-29 Don&#39;t use time.time() or timezone.utcnow() for duration calculations (#12353)
de3b1e687 2020-11-28 Move connection guides to provider documentation packages (#12653)
663259d4b 2020-11-25 Fix AWS DataSync tests failing (#11020)
3fa51f94d 2020-11-24 Add check for duplicates in provider.yaml files (#12578)
ed09915a0 2020-11-23 [AIRFLOW-5115] Bugfix for S3KeySensor failing to accept template_fields (#12389)
370e7d07d 2020-11-21 Fix Python Docstring parameters (#12513)
c34ef853c 2020-11-20 Separate out documentation building per provider (#12444)
008035450 2020-11-18 Update provider READMEs for 1.0.0b2 batch release (#12449)
7ca0b6f12 2020-11-18 Enable Markdownlint rule MD003/heading-style/header-style (#12427) (#12438)

Release 2020.11.23

Commit Committed Subject
4873d9759 2020-11-18 Enable Markdownlint rule MD003/heading-style/header-style (#12427)
ae7cb4a1e 2020-11-17 Update wrong commit hash in backport provider changes (#12390)
6889a333c 2020-11-15 Improvements for operators and hooks ref docs (#12366)
c94b1241a 2020-11-13 Add extra error handling to S3 remote logging (#9908)
7825e8f59 2020-11-13 Docs installation improvements (#12304)
250436d96 2020-11-10 Fix spelling in Python files (#12230)
85a18e13d 2020-11-09 Point at pypi project pages for cross-dependency of provider packages (#12212)
59eb5de78 2020-11-09 Update provider READMEs for up-coming 1.0.0beta1 releases (#12206)
b2a28d159 2020-11-09 Moves provider packages scripts to dev (#12082)
fcb6b00ef 2020-11-08 Add authentication to AWS with Google credentials (#12079)
fb6bddba0 2020-11-07 In AWS Secrets backend, a lookup is optional (#12143)
cf9437d79 2020-11-06 Simplify string expressions (#12123)
41bf172c1 2020-11-04 Simplify string expressions (#12093)
4e8f9cc8d 2020-11-03 Enable Black - Python Auto Formmatter (#9550)
8c42cf1b0 2020-11-03 Use PyUpgrade to use Python 3.6 features (#11447)
5e77a6154 2020-11-02 Docstring fix for S3DeleteBucketOperator (#12049)
822285134 2020-10-29 Add Template Fields to RedshiftToS3Operator & S3ToRedshiftOperator (#11844)
db121f726 2020-10-28 Add truncate table (before copy) option to S3ToRedshiftOperator (#9246)
5a439e84e 2020-10-26 Prepare providers release 0.0.2a1 (#11855)
8afdb6ac6 2020-10-26 Fix spellings (#11825)

Release 2020.10.29

Commit Committed Subject
b680bbc0b 2020-10-24 Generated backport providers readmes/setup for 2020.10.29
6ce855af1 2020-10-24 Fix spelling (#11821)
3934ef224 2020-10-24 Remove redundant builtins imports (#11809)
4c8e033c0 2020-10-24 Fix spelling and grammar (#11814)
483068745 2020-10-24 Use Python 3 style super classes (#11806)
0df60b773 2020-10-23 Add reattach flag to ECSOperator (#10643)
b9d677cdd 2020-10-22 Add type hints to aws provider (#11531)
349b0811c 2020-10-20 Add D200 pydocstyle check (#11688)
674368f66 2020-10-19 Fixes MySQLToS3 float to int conversion (#10437)
0823d46a7 2020-10-16 Add type annotations for AWS operators and hooks (#11434)
16e712971 2020-10-13 Added support for provider packages for Airflow 2.0 (#11487)
d38a0a781 2020-10-12 added type hints for aws cloud formation (#11470)
d305876be 2020-10-12 Remove redundant None provided as default to dict.get() (#11448)
c3e340584 2020-10-11 Change prefix of AwsDynamoDB hook module (#11209)
42a23d16f 2020-10-11 Update MySQLToS3Operator's s3_bucket to template_fields (#10778)
422b61a9d 2020-10-09 Adding ElastiCache Hook for creating, describing and deleting replication groups (#8701)
dd98b2149 2020-10-06 Add acl_policy parameter to GCSToS3Operator (#10804) (#10829)
32b3cfbcf 2020-10-06 Strict type check for all hooks in amazon (#11250)
6d573e8ab 2020-10-03 Add s3 key to template fields for s3/redshift transfer operators (#10890)
0a0e1af80 2020-10-03 Fix Broken Markdown links in Providers README TOC (#11249)

Release 2020.10.5

Commit Committed Subject
ca4238eb4 2020-10-02 Fixed month in backport packages to October (#11242)
5220e4c38 2020-10-02 Prepare Backport release 2020.09.07 (#11238)
00ffedb8c 2020-09-30 Add amazon glacier to GCS transfer operator (#10947)
e3f96ce7a 2020-09-24 Fix incorrect Usage of Optional[bool] (#11138)
f3e87c503 2020-09-22 Add D202 pydocstyle check (#11032)
b61225a88 2020-09-21 Add D204 pydocstyle check (#11031)
2410f592a 2020-09-19 Get Airflow configs with sensitive data from AWS Systems Manager (#11023)
2bf7b7cac 2020-09-19 Add typing to amazon provider EMR (#10910)
9edfcb7ac 2020-09-19 Support extra_args in S3Hook and GCSToS3Operator (#11001)
4e1f3a69d 2020-09-14 [AIRFLOW-10645] Add AWS Secrets Manager Hook (#10655)
e9add7916 2020-09-14 Fix Failing static tests on Master (#10927)
383a118d2 2020-09-14 Add more type annotations to AWS hooks (#10671)
9549274d1 2020-09-09 Upgrade black to 20.8b1 (#10818)
2934220dc 2020-09-08 Always return a list from S3Hook list methods (#10774)
f40ac9b15 2020-09-01 Add placement_strategy option (#9444)
e4878e677 2020-08-31 fix type hints for s3 hook read_key method (#10653)
2ca615cff 2020-08-29 Update Google Cloud branding (#10642)
8969b7185 2020-08-28 Removed bad characters from AWS operator (#10590)
8349061f9 2020-08-26 Improve Docstring for AWS Athena Hook/Operator (#10580)
fdd9b6f65 2020-08-25 Enable Black on Providers Packages (#10543)
3696c34c2 2020-08-24 Fix typo in the word "release" (#10528)
3734876d9 2020-08-24 Implement impersonation in google operators (#10052)
ee7ca128a 2020-08-22 Fix broken Markdown refernces in Providers README (#10483)
c6358045f 2020-08-22 Fixes S3ToRedshift COPY query (#10436)
7c206a82a 2020-08-22 Replace assigment with Augmented assignment (#10468)
27d08b76a 2020-08-21 Amazon SES Hook (#10391)
dea345b05 2020-08-17 Fix AwsGlueJobSensor to stop running after the Glue job finished (#9022)
f6734b3b8 2020-08-12 Enable Sphinx spellcheck for doc generation (#10280)
82f744b87 2020-08-11 Add type annotations to AwsGlueJobHook, RedshiftHook modules (#10286)
19bc97d0c 2020-08-10 Revert "Add Amazon SES hook (#10004)" (#10276)
f06fe616e 2020-08-10 Add Amazon SES hook (#10004)
0c77ea8a3 2020-08-06 Add type annotations to S3 hook module (#10164)
24c8e4c2d 2020-08-06 Changes to all the constructors to remove the args argument (#10163)
9667314b2 2020-08-05 Add correct signatures for operators in amazon provider package (#10167)
000287753 2020-08-03 Improve Typing coverage of amazon/aws/athena (#10025)
53ada6e79 2020-08-03 Add S3KeysUnchangedSensor (#9817)
aeea71274 2020-08-02 Remove args parameter from provider operator constructors (#10097)
2b8dea64e 2020-08-01 Fix typo in Athena sensor retries (#10079)
1508c43ec 2020-07-29 Adding new SageMaker operator for ProcessingJobs (#9594)
7d24b088c 2020-07-25 Stop using start_date in default_args in example_dags (2) (#9985)
8b10a4b35 2020-07-25 Stop using start_date in default_args in example_dags (#9982)
33f0cd265 2020-07-22 apply_default keeps the function signature for mypy (#9784)
e7c87fe45 2020-07-20 Refactor AwsBaseHook._get_credentials (#9878)
2577f9334 2020-07-16 Fix S3FileTransformOperator to support S3 Select transformation only (#8936)
52b6efe1e 2020-07-15 Add option to delete by prefix to S3DeleteObjectsOperator (#9350)
553bb7af7 2020-07-13 Keep functions signatures in decorators (#9786)
2f31b3060 2020-07-08 Get Airflow configs with sensitive data from Secret Backends (#9645)
07b81029e 2020-07-08 Allow AWSAthenaHook to get more than 1000/first page of results (#6075)
564192c16 2020-07-08 Add AWS StepFunctions integrations to the aws provider (#8749)
ecce1ace7 2020-07-08 [AIRFLOW-XXXX] Remove unnecessary docstring in AWSAthenaOperator
a79e2d4c4 2020-07-06 Move provider's log task handlers to the provider package (#9604)
ee20086b8 2020-07-02 Move S3TaskHandler to the AWS provider package (#9602)
40add26d4 2020-06-29 Remove almost all references to airflow.contrib (#9559)
c858babdd 2020-06-26 Remove kwargs from Super calls in AWS Secrets Backends (#9523)
87fdbd070 2020-06-25 Use literal syntax instead of function calls to create data structure (#9516)
c7a454aa3 2020-06-22 Add AWS ECS system test (#8888)
df8efd04f 2020-06-21 Enable & Fix "Docstring Content Issues" PyDocStyle Check (#9460)
e13a14c87 2020-06-21 Enable & Fix Whitespace related PyDocStyle Checks (#9458)
d0e7db402 2020-06-19 Fixed release number for fresh release (#9408)

Release 2020.6.24

Commit Committed Subject
12af6a080 2020-06-19 Final cleanup for 2020.6.23rc1 release preparation (#9404)
992a18c84 2020-06-19 Move MySqlToS3Operator to transfers (#9400)
a60f589aa 2020-06-19 Add MySqlToS3Operator (#9054)
c7e5bce57 2020-06-19 Prepare backport release candidate for 2020.6.23rc1 (#9370)
40bf8f28f 2020-06-18 Detect automatically the lack of reference to the guide in the operator descriptions (#9290)
f6bd817a3 2020-06-16 Introduce 'transfers' packages (#9320)
58a8ec0e4 2020-06-16 AWSBatchOperator <> ClientHook relation changed to composition (#9306)
a80cd25e8 2020-06-15 Close/Flush byte stream in s3 hook load_string and load_bytes (#9211)
ffb857403 2020-06-14 Decrypt secrets from SystemsManagerParameterStoreBackend (#9214)
a69b031f2 2020-06-10 Add S3ToRedshift example dag and system test (#8877)
17adcea83 2020-06-02 Fix handling of subprocess error handling in s3_file_transform and gcs (#9106)
357e11e0c 2020-05-29 Add Delete/Create S3 bucket operators (#8895)
1ed171bfb 2020-05-28 Add script_args for S3FileTransformOperator (#9019)
0b0e4f7a4 2020-05-26 Preparing for RC3 release of backports (#9026)
00642a46d 2020-05-26 Fixed name of 20 remaining wrongly named operators. (#8994)
1d36b0303 2020-05-23 Fix references in docs (#8984)
f946f96da 2020-05-23 Old json boto compat removed from dynamodb_to_s3 operator (#8987)
375d1ca22 2020-05-19 Release candidate 2 for backport packages 2020.05.20 (#8898)
12c5e5d8a 2020-05-17 Prepare release candidate for backport packages (#8891)
f3521fb0e 2020-05-16 Regenerate readme files for backport package release (#8886)
f4edd90a9 2020-05-16 Speed up TestAwsLambdaHook by not actually running a function (#8882)
92585ca4c 2020-05-15 Added automated release notes generation for backport operators (#8807)
85bbab27d 2020-05-15 Add EMR operators howto docs (#8863)
e61b9bb9b 2020-05-13 Add AWS EMR System tests (#8618)
ed3f5131a 2020-05-13 Correctly pass sleep time from AWSAthenaOperator down to the hook. (#8845)
7236862a1 2020-05-12 [AIRFLOW-2310] Enable AWS Glue Job Integration (#6007)
d590e5e76 2020-05-11 Add option to propagate tags in ECSOperator (#8811)
0c3db84c3 2020-05-11 [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor (#7731)
cbebed2b4 2020-05-10 Allow passing backend_kwargs to AWS SSM client (#8802)
c7788a689 2020-05-10 Add imap_attachment_to_s3 example dag and system test (#8669)
ff5b70149 2020-05-07 Add google_api_to_s3_transfer example dags and system tests (#8581)
4421f011e 2020-05-01 Improve template capabilities of EMR job and step operators (#8572)
379a884d6 2020-04-28 fix: aws hook should work without conn id (#8534)
74bc316c5 2020-04-27 [AIRFLOW-4438] Add Gzip compression to S3_hook (#8571)
7ea66a1a9 2020-04-26 Add example DAG for ECSOperator (#8452)
b6434dedf 2020-04-24 [AIRFLOW-7111] Add generate_presigned_url method to S3Hook (#8441)
becedd5af 2020-04-19 Remove unrelated EC2 references in ECSOperator (#8451)
ab1290cb0 2020-04-13 Make launch_type parameter optional (#8248)
87969a350 2020-04-09 [AIRFLOW-6515] Change Log Levels from Info/Warn to Error (#8170)
b46d6c060 2020-04-08 Add support for AWS Secrets Manager as Secrets Backend (#8186)
68d1714f2 2020-04-04 [AIRFLOW-6822] AWS hooks should cache boto3 client (#7541)
8a0240257 2020-03-31 Rename CloudBaseHook to GoogleBaseHook and move it to google.common (#8011)
7239d9a82 2020-03-28 Get Airflow Variables from AWS Systems Manager Parameter Store (#7945)
eb4af4f94 2020-03-28 Make BaseSecretsBackend.build_path generic (#7948)
438da7241 2020-03-28 [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent (#7891)
686d7d50b 2020-03-25 Standardize SecretBackend class names (#7846)
eef87b995 2020-03-23 [AIRFLOW-7105] Unify Secrets Backend method interfaces (#7830)
5648dfbc3 2020-03-23 Add missing call to Super class in 'amazon', 'cloudant & 'databricks' providers (#7827)
a36002412 2020-03-19 [AIRFLOW-5705] Make AwsSsmSecretsBackend consistent with VaultBackend (#7753)
2a54512d7 2020-03-17 [AIRFLOW-5705] Fix bugs in AWS SSM Secrets Backend (#7745)
a8b5fc74d 2020-03-16 [AIRFLOW-4175] S3Hook load_file should support ACL policy paramete (#7733)
e31e9ddd2 2020-03-14 [AIRFLOW-5705] Add secrets backend and support for AWS SSM (#6376)
3bb60afc7 2020-03-13 [AIRFLOW-6975] Base AWSHook AssumeRoleWithSAML (#7619)
c0c5f11ad 2020-03-12 [AIRFLOW-6884] Make SageMakerTrainingOperator idempotent (#7598)
b7cdda1c6 2020-03-10 [AIRFLOW-4438] Add Gzip compression to S3_hook (#7680)
42eef3821 2020-03-07 [AIRFLOW-6877] Add cross-provider dependencies as extras (#7506)
9a94ab246 2020-03-01 [AIRFLOW-6962] Fix compeleted to completed (#7600)
1b38f6d9b 2020-02-29 [AIRFLOW-5908] Add download_file to S3 Hook (#6577)
3ea3e1a2b 2020-02-26 [AIRFLOW-6824] EMRAddStepsOperator problem with multi-step XCom (#7443)
6eaa7e3b1 2020-02-25 [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook (#6574)
3320e432a 2020-02-24 [AIRFLOW-6817] Lazy-load airflow.DAG to keep user-facing API untouched (#7517)
7d0e7122d 2020-02-24 [AIRFLOW-6830] Add Subject/MessageAttributes to SNS hook and operator (#7451)
4d03e33c1 2020-02-22 [AIRFLOW-6817] remove imports from airflow/__init__.py, replaced implicit imports with explicit imports, added entry to UPDATING.MD - squashed/rebased (#7456)
47a922b86 2020-02-21 [AIRFLOW-6854] Fix missing typing_extensions on python 3.8 (#7474)
9cbd7de6d 2020-02-18 [AIRFLOW-6792] Remove _operator/_hook/_sensor in providers package and add tests (#7412)
58c3542ed 2020-02-12 [AIRFLOW-5231] Fix S3Hook.delete_objects method (#7375)
b7aa778b3 2020-02-10 [AIRFLOW-6767] Correct name for default Athena workgroup (#7394)
9282185e6 2020-02-09 [AIRFLOW-6761] Fix WorkGroup param in AWSAthenaHook (#7386)
94fccca97 2020-02-04 [AIRFLOW-XXXX] Add pre-commit check for utf-8 file encoding (#7347)
f3ad5cf61 2020-02-03 [AIRFLOW-4681] Make sensors module pylint compatible (#7309)
88e40c714 2020-02-03 [AIRFLOW-6716] Fix AWS Datasync Example DAG (#7339)
a311d3d82 2020-02-03 [AIRFLOW-6718] Fix more occurrences of utils.dates.days_ago (#7341)
cb766b05b 2020-02-03 [AIRFLOW-XXXX] Fix Static Checks on CI (#7342)
97a429f9d 2020-02-02 [AIRFLOW-6714] Remove magic comments about UTF-8 (#7338)
7527eddc5 2020-02-02 [AIRFLOW-4364] Make all code in airflow/providers/amazon pylint compatible (#7336)
cf141506a 2020-02-02 [AIRFLOW-6708] Set unique logger names (#7330)
63aa3db88 2020-02-02 [AIRFLOW-6258] Add CloudFormation operators to AWS providers (#6824)
af4157fde 2020-02-02 [AIRFLOW-6672] AWS DataSync - better logging of error message (#7288)
373c6aa4a 2020-01-30 [AIRFLOW-6682] Move GCP classes to providers package (#7295)
83c037873 2020-01-30 [AIRFLOW-6674] Move example_dags in accordance with AIP-21 (#7287)
1988a97e8 2020-01-28 [AIRFLOW-6659] Move AWS Transfer operators to providers package (#7274)
ab10443e9 2020-01-28 [AIRFLOW-6424] Added a operator to modify EMR cluster (#7213)
40246132a 2020-01-28 [AIRFLOW-6654] AWS DataSync - bugfix when creating locations (#7270)
82c0e5aff 2020-01-28 [AIRFLOW-6655] Move AWS classes to providers (#7271)
599e4791c 2020-01-18 [AIRFLOW-6541] Use EmrJobFlowSensor for other states (#7146)
c319e81ca 2020-01-17 [AIRFLOW-6572] Move AWS classes to providers.amazon.aws package (#7178)
941a07057 2020-01-15 [AIRFLOW-6570] Add dag tag for all example dag (#7176)
78d8fe694 2020-01-08 [AIRFLOW-6245] Add custom waiters for AWS batch jobs (#6811)
e0b022725 2019-12-28 [AIRFLOW-6319] Add support for AWS Athena workgroups (#6871)
57da45685 2019-12-24 [AIRFLOW-6333] Bump Pylint to 2.4.4 & fix/disable new checks (#6888)
cf647c27e 2019-12-18 [AIRFLOW-6038] AWS DataSync reworked (#6773)
7502cad28 2019-12-17 [AIRFLOW-6206] Move and rename AWS batch operator [AIP-21] (#6764)
c4c635df6 2019-12-17 [AIRFLOW-6083] Adding ability to pass custom configuration to lambda client. (#6678)
4fb498f87 2019-12-09 [AIRFLOW-6072] aws_hook: Outbound http proxy setting and other enhancements (#6686)
a1e2f8635 2019-11-20 [AIRFLOW-6021] Replace list literal with list constructor (#6617)
baae14084 2019-11-19 [AIRFLOW-5781] AIP-21 Migrate AWS Kinesis to /providers/amazon/aws (#6588)
504cfbac1 2019-11-18 [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure (#6539)
992f0e3ac 2019-11-12 AIRFLOW-5824: AWS DataSync Hook and Operators added (#6512)
c015eb2f6 2019-11-10 [AIRFLOW-5786] Migrate AWS SNS to /providers/amazon/aws (#6502)
3d76fb4bf 2019-11-09 [AIRFLOW-5782] Migrate AWS Lambda to /providers/amazon/aws [AIP-21] (#6518)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Built Distribution

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page