apache-airflow-backport-providers-amazon 2020.6.24

Back-ported airflow.providers.amazon.* package for Airflow 1.10.*

Package apache-airflow-backport-providers-amazon

Release: 2020.6.24

Backport package

This is a backport providers package for amazon provider. All classes for this provider package are in airflow.providers.amazon python package.

Only Python 3.6+ is supported for this backport package.

While Airflow 1.10.* continues to support Python 2.7+ - you need to upgrade python to 3.6+ if you want to use this backport package.

Installation

You can install this package on top of an existing airflow 1.10.* installation via pip install apache-airflow-backport-providers-amazon

Compatibility

For full compatibility and test status of the backport packages check Airflow Backport Package Compatibility

PIP requirements

PIP package	Version required
boto3	>=1.12.0,<2.0.0
watchtower	~=0.7.3

Cross provider package dependencies

Those are dependencies that might be needed in order to use all the features of the package. You need to install the specified backport providers package in order to use them.

You can install such cross-provider dependencies when installing from PyPI. For example:

pip install apache-airflow-backport-providers-amazon[apache.hive]

Dependent package	Extra
apache-airflow-backport-providers-apache-hive	apache.hive
apache-airflow-backport-providers-google	google
apache-airflow-backport-providers-imap	imap
apache-airflow-backport-providers-mongo	mongo
apache-airflow-backport-providers-mysql	mysql
apache-airflow-backport-providers-postgres	postgres
apache-airflow-backport-providers-ssh	ssh

Provider classes summary

In Airflow 2.0, all operators, transfers, hooks, sensors, secrets for the amazon provider are in the airflow.providers.amazon package. You can read more about the naming conventions used in Naming conventions for provider packages

Operators

New operators

New Airflow 2.0 operators: airflow.providers.amazon package
aws.operators.cloud_formation.CloudFormationCreateStackOperator
aws.operators.cloud_formation.CloudFormationDeleteStackOperator
aws.operators.datasync.AWSDataSyncOperator
aws.operators.ec2_start_instance.EC2StartInstanceOperator
aws.operators.ec2_stop_instance.EC2StopInstanceOperator
aws.operators.emr_modify_cluster.EmrModifyClusterOperator
aws.operators.glue.AwsGlueJobOperator
aws.operators.s3_bucket.S3CreateBucketOperator
aws.operators.s3_bucket.S3DeleteBucketOperator
aws.operators.s3_file_transform.S3FileTransformOperator

Moved operators

Airflow 2.0 operators: airflow.providers.amazon package	Airflow 1.10.* previous location (usually airflow.contrib)
aws.operators.athena.AWSAthenaOperator	contrib.operators.aws_athena_operator.AWSAthenaOperator
aws.operators.batch.AwsBatchOperator	contrib.operators.awsbatch_operator.AWSBatchOperator
aws.operators.ecs.ECSOperator	contrib.operators.ecs_operator.ECSOperator
aws.operators.emr_add_steps.EmrAddStepsOperator	contrib.operators.emr_add_steps_operator.EmrAddStepsOperator
aws.operators.emr_create_job_flow.EmrCreateJobFlowOperator	contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator
aws.operators.emr_terminate_job_flow.EmrTerminateJobFlowOperator	contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator
aws.operators.s3_copy_object.S3CopyObjectOperator	contrib.operators.s3_copy_object_operator.S3CopyObjectOperator
aws.operators.s3_delete_objects.S3DeleteObjectsOperator	contrib.operators.s3_delete_objects_operator.S3DeleteObjectsOperator
aws.operators.s3_list.S3ListOperator	contrib.operators.s3_list_operator.S3ListOperator
aws.operators.sagemaker_base.SageMakerBaseOperator	contrib.operators.sagemaker_base_operator.SageMakerBaseOperator
aws.operators.sagemaker_endpoint.SageMakerEndpointOperator	contrib.operators.sagemaker_endpoint_operator.SageMakerEndpointOperator
aws.operators.sagemaker_endpoint_config.SageMakerEndpointConfigOperator	contrib.operators.sagemaker_endpoint_config_operator.SageMakerEndpointConfigOperator
aws.operators.sagemaker_model.SageMakerModelOperator	contrib.operators.sagemaker_model_operator.SageMakerModelOperator
aws.operators.sagemaker_training.SageMakerTrainingOperator	contrib.operators.sagemaker_training_operator.SageMakerTrainingOperator
aws.operators.sagemaker_transform.SageMakerTransformOperator	contrib.operators.sagemaker_transform_operator.SageMakerTransformOperator
aws.operators.sagemaker_tuning.SageMakerTuningOperator	contrib.operators.sagemaker_tuning_operator.SageMakerTuningOperator
aws.operators.sns.SnsPublishOperator	contrib.operators.sns_publish_operator.SnsPublishOperator
aws.operators.sqs.SQSPublishOperator	contrib.operators.aws_sqs_publish_operator.SQSPublishOperator

New transfer operators

New Airflow 2.0 transfers: airflow.providers.amazon package
aws.transfers.mysql_to_s3.MySQLToS3Operator

Moved transfer operators

Airflow 2.0 transfers: airflow.providers.amazon package	Airflow 1.10.* previous location (usually airflow.contrib)
aws.transfers.dynamodb_to_s3.DynamoDBToS3Operator	contrib.operators.dynamodb_to_s3.DynamoDBToS3Operator
aws.transfers.gcs_to_s3.GCSToS3Operator	operators.gcs_to_s3.GCSToS3Operator
aws.transfers.google_api_to_s3.GoogleApiToS3Operator	operators.google_api_to_s3_transfer.GoogleApiToS3Transfer
aws.transfers.hive_to_dynamodb.HiveToDynamoDBOperator	contrib.operators.hive_to_dynamodb.HiveToDynamoDBOperator
aws.transfers.imap_attachment_to_s3.ImapAttachmentToS3Operator	contrib.operators.imap_attachment_to_s3_operator.ImapAttachmentToS3Operator
aws.transfers.mongo_to_s3.MongoToS3Operator	contrib.operators.mongo_to_s3.MongoToS3Operator
aws.transfers.redshift_to_s3.RedshiftToS3Operator	operators.redshift_to_s3_operator.RedshiftToS3Transfer
aws.transfers.s3_to_redshift.S3ToRedshiftOperator	operators.s3_to_redshift_operator.S3ToRedshiftTransfer
aws.transfers.s3_to_sftp.S3ToSFTPOperator	contrib.operators.s3_to_sftp_operator.S3ToSFTPOperator
aws.transfers.sftp_to_s3.SFTPToS3Operator	contrib.operators.sftp_to_s3_operator.SFTPToS3Operator

Sensors

New sensors

New Airflow 2.0 sensors: airflow.providers.amazon package
aws.sensors.cloud_formation.CloudFormationCreateStackSensor
aws.sensors.cloud_formation.CloudFormationDeleteStackSensor
aws.sensors.ec2_instance_state.EC2InstanceStateSensor
aws.sensors.glue.AwsGlueJobSensor
aws.sensors.redshift.AwsRedshiftClusterSensor
aws.sensors.sagemaker_training.SageMakerTrainingSensor

Moved sensors

Airflow 2.0 sensors: airflow.providers.amazon package	Airflow 1.10.* previous location (usually airflow.contrib)
aws.sensors.athena.AthenaSensor	contrib.sensors.aws_athena_sensor.AthenaSensor
aws.sensors.emr_base.EmrBaseSensor	contrib.sensors.emr_base_sensor.EmrBaseSensor
aws.sensors.emr_job_flow.EmrJobFlowSensor	contrib.sensors.emr_job_flow_sensor.EmrJobFlowSensor
aws.sensors.emr_step.EmrStepSensor	contrib.sensors.emr_step_sensor.EmrStepSensor
aws.sensors.glue_catalog_partition.AwsGlueCatalogPartitionSensor	contrib.sensors.aws_glue_catalog_partition_sensor.AwsGlueCatalogPartitionSensor
aws.sensors.s3_key.S3KeySensor	sensors.s3_key_sensor.S3KeySensor
aws.sensors.s3_prefix.S3PrefixSensor	sensors.s3_prefix_sensor.S3PrefixSensor
aws.sensors.sagemaker_base.SageMakerBaseSensor	contrib.sensors.sagemaker_base_sensor.SageMakerBaseSensor
aws.sensors.sagemaker_endpoint.SageMakerEndpointSensor	contrib.sensors.sagemaker_endpoint_sensor.SageMakerEndpointSensor
aws.sensors.sagemaker_transform.SageMakerTransformSensor	contrib.sensors.sagemaker_transform_sensor.SageMakerTransformSensor
aws.sensors.sagemaker_tuning.SageMakerTuningSensor	contrib.sensors.sagemaker_tuning_sensor.SageMakerTuningSensor
aws.sensors.sqs.SQSSensor	contrib.sensors.aws_sqs_sensor.SQSSensor

Hooks

New hooks

New Airflow 2.0 hooks: airflow.providers.amazon package
aws.hooks.batch_client.AwsBatchClientHook
aws.hooks.batch_waiters.AwsBatchWaitersHook
aws.hooks.cloud_formation.AWSCloudFormationHook
aws.hooks.ec2.EC2Hook
aws.hooks.glue.AwsGlueJobHook
aws.hooks.kinesis.AwsFirehoseHook
aws.hooks.redshift.RedshiftHook

Moved hooks

Airflow 2.0 hooks: airflow.providers.amazon package	Airflow 1.10.* previous location (usually airflow.contrib)
aws.hooks.athena.AWSAthenaHook	contrib.hooks.aws_athena_hook.AWSAthenaHook
aws.hooks.aws_dynamodb.AwsDynamoDBHook	contrib.hooks.aws_dynamodb_hook.AwsDynamoDBHook
aws.hooks.base_aws.AwsBaseHook	contrib.hooks.aws_hook.AwsHook
aws.hooks.datasync.AWSDataSyncHook	contrib.hooks.aws_datasync_hook.AWSDataSyncHook
aws.hooks.emr.EmrHook	contrib.hooks.emr_hook.EmrHook
aws.hooks.glue_catalog.AwsGlueCatalogHook	contrib.hooks.aws_glue_catalog_hook.AwsGlueCatalogHook
aws.hooks.lambda_function.AwsLambdaHook	contrib.hooks.aws_lambda_hook.AwsLambdaHook
aws.hooks.logs.AwsLogsHook	contrib.hooks.aws_logs_hook.AwsLogsHook
aws.hooks.s3.S3Hook	hooks.S3_hook.S3Hook
aws.hooks.sagemaker.SageMakerHook	contrib.hooks.sagemaker_hook.SageMakerHook
aws.hooks.sns.AwsSnsHook	contrib.hooks.aws_sns_hook.AwsSnsHook
aws.hooks.sqs.SQSHook	contrib.hooks.aws_sqs_hook.SQSHook

Secrets

Moved secrets

Airflow 2.0 protocols: airflow.providers.amazon package	Airflow 1.10.* previous location (usually airflow.contrib)
aws.secrets.secrets_manager.SecretsManagerBackend	contrib.secrets.aws_secrets_manager.SecretsManagerBackend
aws.secrets.systems_manager.SystemsManagerParameterStoreBackend	contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend

Releases

Release 2020.6.24

Commit	Committed	Subject
12af6a080	2020-06-19	Final cleanup for 2020.6.23rc1 release preparation (#9404)
992a18c84	2020-06-19	Move MySqlToS3Operator to transfers (#9400)
a60f589aa	2020-06-19	Add MySqlToS3Operator (#9054)
c7e5bce57	2020-06-19	Prepare backport release candidate for 2020.6.23rc1 (#9370)
40bf8f28f	2020-06-18	Detect automatically the lack of reference to the guide in the operator descriptions (#9290)
f6bd817a3	2020-06-16	Introduce 'transfers' packages (#9320)
58a8ec0e4	2020-06-16	AWSBatchOperator <> ClientHook relation changed to composition (#9306)
a80cd25e8	2020-06-15	Close/Flush byte stream in s3 hook load_string and load_bytes (#9211)
ffb857403	2020-06-14	Decrypt secrets from SystemsManagerParameterStoreBackend (#9214)
a69b031f2	2020-06-10	Add S3ToRedshift example dag and system test (#8877)
17adcea83	2020-06-02	Fix handling of subprocess error handling in s3_file_transform and gcs (#9106)
357e11e0c	2020-05-29	Add Delete/Create S3 bucket operators (#8895)
1ed171bfb	2020-05-28	Add script_args for S3FileTransformOperator (#9019)
0b0e4f7a4	2020-05-26	Preparing for RC3 relase of backports (#9026)
00642a46d	2020-05-26	Fixed name of 20 remaining wrongly named operators. (#8994)
1d36b0303	2020-05-23	Fix references in docs (#8984)
f946f96da	2020-05-23	Old json boto compat removed from dynamodb_to_s3 operator (#8987)
375d1ca22	2020-05-19	Release candidate 2 for backport packages 2020.05.20 (#8898)
12c5e5d8a	2020-05-17	Prepare release candidate for backport packages (#8891)
f3521fb0e	2020-05-16	Regenerate readme files for backport package release (#8886)
f4edd90a9	2020-05-16	Speed up TestAwsLambdaHook by not actually running a function (#8882)
92585ca4c	2020-05-15	Added automated release notes generation for backport operators (#8807)
85bbab27d	2020-05-15	Add EMR operators howto docs (#8863)
e61b9bb9b	2020-05-13	Add AWS EMR System tests (#8618)
ed3f5131a	2020-05-13	Correctly pass sleep time from AWSAthenaOperator down to the hook. (#8845)
7236862a1	2020-05-12	[AIRFLOW-2310] Enable AWS Glue Job Integration (#6007)
d590e5e76	2020-05-11	Add option to propagate tags in ECSOperator (#8811)
0c3db84c3	2020-05-11	[AIRFLOW-7068] Create EC2 Hook, Operator and Sensor (#7731)
cbebed2b4	2020-05-10	Allow passing backend_kwargs to AWS SSM client (#8802)
c7788a689	2020-05-10	Add imap_attachment_to_s3 example dag and system test (#8669)
ff5b70149	2020-05-07	Add google_api_to_s3_transfer example dags and system tests (#8581)
4421f011e	2020-05-01	Improve template capabilities of EMR job and step operators (#8572)
379a884d6	2020-04-28	fix: aws hook should work without conn id (#8534)
74bc316c5	2020-04-27	[AIRFLOW-4438] Add Gzip compression to S3_hook (#8571)
7ea66a1a9	2020-04-26	Add example DAG for ECSOperator (#8452)
b6434dedf	2020-04-24	[AIRFLOW-7111] Add generate_presigned_url method to S3Hook (#8441)
becedd5af	2020-04-19	Remove unrelated EC2 references in ECSOperator (#8451)
ab1290cb0	2020-04-13	Make launch_type parameter optional (#8248)
87969a350	2020-04-09	[AIRFLOW-6515] Change Log Levels from Info/Warn to Error (#8170)
b46d6c060	2020-04-08	Add support for AWS Secrets Manager as Secrets Backend (#8186)
68d1714f2	2020-04-04	[AIRFLOW-6822] AWS hooks should cache boto3 client (#7541)
8a0240257	2020-03-31	Rename CloudBaseHook to GoogleBaseHook and move it to google.common (#8011)
7239d9a82	2020-03-28	Get Airflow Variables from AWS Systems Manager Parameter Store (#7945)
eb4af4f94	2020-03-28	Make BaseSecretsBackend.build_path generic (#7948)
438da7241	2020-03-28	[AIRFLOW-5825] SageMakerEndpointOperator is not idempotent (#7891)
686d7d50b	2020-03-25	Standardize SecretBackend class names (#7846)
eef87b995	2020-03-23	[AIRFLOW-7105] Unify Secrets Backend method interfaces (#7830)
5648dfbc3	2020-03-23	Add missing call to Super class in 'amazon', 'cloudant & 'databricks' providers (#7827)
a36002412	2020-03-19	[AIRFLOW-5705] Make AwsSsmSecretsBackend consistent with VaultBackend (#7753)
2a54512d7	2020-03-17	[AIRFLOW-5705] Fix bugs in AWS SSM Secrets Backend (#7745)
a8b5fc74d	2020-03-16	[AIRFLOW-4175] S3Hook load_file should support ACL policy paramete (#7733)
e31e9ddd2	2020-03-14	[AIRFLOW-5705] Add secrets backend and support for AWS SSM (#6376)
3bb60afc7	2020-03-13	[AIRFLOW-6975] Base AWSHook AssumeRoleWithSAML (#7619)
c0c5f11ad	2020-03-12	[AIRFLOW-6884] Make SageMakerTrainingOperator idempotent (#7598)
b7cdda1c6	2020-03-10	[AIRFLOW-4438] Add Gzip compression to S3_hook (#7680)
42eef3821	2020-03-07	[AIRFLOW-6877] Add cross-provider dependencies as extras (#7506)
9a94ab246	2020-03-01	[AIRFLOW-6962] Fix compeleted to completed (#7600)
1b38f6d9b	2020-02-29	[AIRFLOW-5908] Add download_file to S3 Hook (#6577)
3ea3e1a2b	2020-02-26	[AIRFLOW-6824] EMRAddStepsOperator problem with multi-step XCom (#7443)
6eaa7e3b1	2020-02-25	[AIRFLOW-5924] Automatically unify bucket name and key in S3Hook (#6574)
3320e432a	2020-02-24	[AIRFLOW-6817] Lazy-load airflow.DAG to keep user-facing API untouched (#7517)
7d0e7122d	2020-02-24	[AIRFLOW-6830] Add Subject/MessageAttributes to SNS hook and operator (#7451)
4d03e33c1	2020-02-22	[AIRFLOW-6817] remove imports from airflow/__init__.py, replaced implicit imports with explicit imports, added entry to UPDATING.MD - squashed/rebased (#7456)
47a922b86	2020-02-21	[AIRFLOW-6854] Fix missing typing_extensions on python 3.8 (#7474)
9cbd7de6d	2020-02-18	[AIRFLOW-6792] Remove _operator/_hook/_sensor in providers package and add tests (#7412)
58c3542ed	2020-02-12	[AIRFLOW-5231] Fix S3Hook.delete_objects method (#7375)
b7aa778b3	2020-02-10	[AIRFLOW-6767] Correct name for default Athena workgroup (#7394)
9282185e6	2020-02-09	[AIRFLOW-6761] Fix WorkGroup param in AWSAthenaHook (#7386)
94fccca97	2020-02-04	[AIRFLOW-XXXX] Add pre-commit check for utf-8 file encoding (#7347)
f3ad5cf61	2020-02-03	[AIRFLOW-4681] Make sensors module pylint compatible (#7309)
88e40c714	2020-02-03	[AIRFLOW-6716] Fix AWS Datasync Example DAG (#7339)
a311d3d82	2020-02-03	[AIRFLOW-6718] Fix more occurrences of utils.dates.days_ago (#7341)
cb766b05b	2020-02-03	[AIRFLOW-XXXX] Fix Static Checks on CI (#7342)
97a429f9d	2020-02-02	[AIRFLOW-6714] Remove magic comments about UTF-8 (#7338)
7527eddc5	2020-02-02	[AIRFLOW-4364] Make all code in airflow/providers/amazon pylint compatible (#7336)
cf141506a	2020-02-02	[AIRFLOW-6708] Set unique logger names (#7330)
63aa3db88	2020-02-02	[AIRFLOW-6258] Add CloudFormation operators to AWS providers (#6824)
af4157fde	2020-02-02	[AIRFLOW-6672] AWS DataSync - better logging of error message (#7288)
373c6aa4a	2020-01-30	[AIRFLOW-6682] Move GCP classes to providers package (#7295)
83c037873	2020-01-30	[AIRFLOW-6674] Move example_dags in accordance with AIP-21 (#7287)
1988a97e8	2020-01-28	[AIRFLOW-6659] Move AWS Transfer operators to providers package (#7274)
ab10443e9	2020-01-28	[AIRFLOW-6424] Added a operator to modify EMR cluster (#7213)
40246132a	2020-01-28	[AIRFLOW-6654] AWS DataSync - bugfix when creating locations (#7270)
82c0e5aff	2020-01-28	[AIRFLOW-6655] Move AWS classes to providers (#7271)
599e4791c	2020-01-18	[AIRFLOW-6541] Use EmrJobFlowSensor for other states (#7146)
c319e81ca	2020-01-17	[AIRFLOW-6572] Move AWS classes to providers.amazon.aws package (#7178)
941a07057	2020-01-15	[AIRFLOW-6570] Add dag tag for all example dag (#7176)
78d8fe694	2020-01-08	[AIRFLOW-6245] Add custom waiters for AWS batch jobs (#6811)
e0b022725	2019-12-28	[AIRFLOW-6319] Add support for AWS Athena workgroups (#6871)
57da45685	2019-12-24	[AIRFLOW-6333] Bump Pylint to 2.4.4 & fix/disable new checks (#6888)
cf647c27e	2019-12-18	[AIRFLOW-6038] AWS DataSync reworked (#6773)
7502cad28	2019-12-17	[AIRFLOW-6206] Move and rename AWS batch operator [AIP-21] (#6764)
c4c635df6	2019-12-17	[AIRFLOW-6083] Adding ability to pass custom configuration to lambda client. (#6678)
4fb498f87	2019-12-09	[AIRFLOW-6072] aws_hook: Outbound http proxy setting and other enhancements (#6686)
a1e2f8635	2019-11-20	[AIRFLOW-6021] Replace list literal with list constructor (#6617)
baae14084	2019-11-19	[AIRFLOW-5781] AIP-21 Migrate AWS Kinesis to /providers/amazon/aws (#6588)
504cfbac1	2019-11-18	[AIRFLOW-5783] AIP-21 Move aws redshift into providers structure (#6539)
992f0e3ac	2019-11-12	AIRFLOW-5824: AWS DataSync Hook and Operators added (#6512)
c015eb2f6	2019-11-10	[AIRFLOW-5786] Migrate AWS SNS to /providers/amazon/aws (#6502)
3d76fb4bf	2019-11-09	[AIRFLOW-5782] Migrate AWS Lambda to /providers/amazon/aws [AIP-21] (#6518)