Skip to main content

Back-ported airflow.providers.amazon.* package for Airflow 1.10.*

Project description

Package apache-airflow-backport-providers-amazon

Release: 2020.6.24

Backport package

This is a backport providers package for amazon provider. All classes for this provider package are in airflow.providers.amazon python package.

Only Python 3.6+ is supported for this backport package.

While Airflow 1.10.* continues to support Python 2.7+ - you need to upgrade python to 3.6+ if you want to use this backport package.

Installation

You can install this package on top of an existing airflow 1.10.* installation via pip install apache-airflow-backport-providers-amazon

Compatibility

For full compatibility and test status of the backport packages check Airflow Backport Package Compatibility

PIP requirements

PIP package Version required
boto3 >=1.12.0,<2.0.0
watchtower ~=0.7.3

Cross provider package dependencies

Those are dependencies that might be needed in order to use all the features of the package. You need to install the specified backport providers package in order to use them.

You can install such cross-provider dependencies when installing from PyPI. For example:

pip install apache-airflow-backport-providers-amazon[apache.hive]
Dependent package Extra
apache-airflow-backport-providers-apache-hive apache.hive
apache-airflow-backport-providers-google google
apache-airflow-backport-providers-imap imap
apache-airflow-backport-providers-mongo mongo
apache-airflow-backport-providers-mysql mysql
apache-airflow-backport-providers-postgres postgres
apache-airflow-backport-providers-ssh ssh

Provider classes summary

In Airflow 2.0, all operators, transfers, hooks, sensors, secrets for the amazon provider are in the airflow.providers.amazon package. You can read more about the naming conventions used in Naming conventions for provider packages

Operators

New operators

New Airflow 2.0 operators: airflow.providers.amazon package
aws.operators.cloud_formation.CloudFormationCreateStackOperator
aws.operators.cloud_formation.CloudFormationDeleteStackOperator
aws.operators.datasync.AWSDataSyncOperator
aws.operators.ec2_start_instance.EC2StartInstanceOperator
aws.operators.ec2_stop_instance.EC2StopInstanceOperator
aws.operators.emr_modify_cluster.EmrModifyClusterOperator
aws.operators.glue.AwsGlueJobOperator
aws.operators.s3_bucket.S3CreateBucketOperator
aws.operators.s3_bucket.S3DeleteBucketOperator
aws.operators.s3_file_transform.S3FileTransformOperator

Moved operators

Airflow 2.0 operators: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.operators.athena.AWSAthenaOperator contrib.operators.aws_athena_operator.AWSAthenaOperator
aws.operators.batch.AwsBatchOperator contrib.operators.awsbatch_operator.AWSBatchOperator
aws.operators.ecs.ECSOperator contrib.operators.ecs_operator.ECSOperator
aws.operators.emr_add_steps.EmrAddStepsOperator contrib.operators.emr_add_steps_operator.EmrAddStepsOperator
aws.operators.emr_create_job_flow.EmrCreateJobFlowOperator contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator
aws.operators.emr_terminate_job_flow.EmrTerminateJobFlowOperator contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator
aws.operators.s3_copy_object.S3CopyObjectOperator contrib.operators.s3_copy_object_operator.S3CopyObjectOperator
aws.operators.s3_delete_objects.S3DeleteObjectsOperator contrib.operators.s3_delete_objects_operator.S3DeleteObjectsOperator
aws.operators.s3_list.S3ListOperator contrib.operators.s3_list_operator.S3ListOperator
aws.operators.sagemaker_base.SageMakerBaseOperator contrib.operators.sagemaker_base_operator.SageMakerBaseOperator
aws.operators.sagemaker_endpoint.SageMakerEndpointOperator contrib.operators.sagemaker_endpoint_operator.SageMakerEndpointOperator
aws.operators.sagemaker_endpoint_config.SageMakerEndpointConfigOperator contrib.operators.sagemaker_endpoint_config_operator.SageMakerEndpointConfigOperator
aws.operators.sagemaker_model.SageMakerModelOperator contrib.operators.sagemaker_model_operator.SageMakerModelOperator
aws.operators.sagemaker_training.SageMakerTrainingOperator contrib.operators.sagemaker_training_operator.SageMakerTrainingOperator
aws.operators.sagemaker_transform.SageMakerTransformOperator contrib.operators.sagemaker_transform_operator.SageMakerTransformOperator
aws.operators.sagemaker_tuning.SageMakerTuningOperator contrib.operators.sagemaker_tuning_operator.SageMakerTuningOperator
aws.operators.sns.SnsPublishOperator contrib.operators.sns_publish_operator.SnsPublishOperator
aws.operators.sqs.SQSPublishOperator contrib.operators.aws_sqs_publish_operator.SQSPublishOperator

New transfer operators

New Airflow 2.0 transfers: airflow.providers.amazon package
aws.transfers.mysql_to_s3.MySQLToS3Operator

Moved transfer operators

Airflow 2.0 transfers: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.transfers.dynamodb_to_s3.DynamoDBToS3Operator contrib.operators.dynamodb_to_s3.DynamoDBToS3Operator
aws.transfers.gcs_to_s3.GCSToS3Operator operators.gcs_to_s3.GCSToS3Operator
aws.transfers.google_api_to_s3.GoogleApiToS3Operator operators.google_api_to_s3_transfer.GoogleApiToS3Transfer
aws.transfers.hive_to_dynamodb.HiveToDynamoDBOperator contrib.operators.hive_to_dynamodb.HiveToDynamoDBOperator
aws.transfers.imap_attachment_to_s3.ImapAttachmentToS3Operator contrib.operators.imap_attachment_to_s3_operator.ImapAttachmentToS3Operator
aws.transfers.mongo_to_s3.MongoToS3Operator contrib.operators.mongo_to_s3.MongoToS3Operator
aws.transfers.redshift_to_s3.RedshiftToS3Operator operators.redshift_to_s3_operator.RedshiftToS3Transfer
aws.transfers.s3_to_redshift.S3ToRedshiftOperator operators.s3_to_redshift_operator.S3ToRedshiftTransfer
aws.transfers.s3_to_sftp.S3ToSFTPOperator contrib.operators.s3_to_sftp_operator.S3ToSFTPOperator
aws.transfers.sftp_to_s3.SFTPToS3Operator contrib.operators.sftp_to_s3_operator.SFTPToS3Operator

Sensors

New sensors

New Airflow 2.0 sensors: airflow.providers.amazon package
aws.sensors.cloud_formation.CloudFormationCreateStackSensor
aws.sensors.cloud_formation.CloudFormationDeleteStackSensor
aws.sensors.ec2_instance_state.EC2InstanceStateSensor
aws.sensors.glue.AwsGlueJobSensor
aws.sensors.redshift.AwsRedshiftClusterSensor
aws.sensors.sagemaker_training.SageMakerTrainingSensor

Moved sensors

Airflow 2.0 sensors: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.sensors.athena.AthenaSensor contrib.sensors.aws_athena_sensor.AthenaSensor
aws.sensors.emr_base.EmrBaseSensor contrib.sensors.emr_base_sensor.EmrBaseSensor
aws.sensors.emr_job_flow.EmrJobFlowSensor contrib.sensors.emr_job_flow_sensor.EmrJobFlowSensor
aws.sensors.emr_step.EmrStepSensor contrib.sensors.emr_step_sensor.EmrStepSensor
aws.sensors.glue_catalog_partition.AwsGlueCatalogPartitionSensor contrib.sensors.aws_glue_catalog_partition_sensor.AwsGlueCatalogPartitionSensor
aws.sensors.s3_key.S3KeySensor sensors.s3_key_sensor.S3KeySensor
aws.sensors.s3_prefix.S3PrefixSensor sensors.s3_prefix_sensor.S3PrefixSensor
aws.sensors.sagemaker_base.SageMakerBaseSensor contrib.sensors.sagemaker_base_sensor.SageMakerBaseSensor
aws.sensors.sagemaker_endpoint.SageMakerEndpointSensor contrib.sensors.sagemaker_endpoint_sensor.SageMakerEndpointSensor
aws.sensors.sagemaker_transform.SageMakerTransformSensor contrib.sensors.sagemaker_transform_sensor.SageMakerTransformSensor
aws.sensors.sagemaker_tuning.SageMakerTuningSensor contrib.sensors.sagemaker_tuning_sensor.SageMakerTuningSensor
aws.sensors.sqs.SQSSensor contrib.sensors.aws_sqs_sensor.SQSSensor

Hooks

New hooks

New Airflow 2.0 hooks: airflow.providers.amazon package
aws.hooks.batch_client.AwsBatchClientHook
aws.hooks.batch_waiters.AwsBatchWaitersHook
aws.hooks.cloud_formation.AWSCloudFormationHook
aws.hooks.ec2.EC2Hook
aws.hooks.glue.AwsGlueJobHook
aws.hooks.kinesis.AwsFirehoseHook
aws.hooks.redshift.RedshiftHook

Moved hooks

Airflow 2.0 hooks: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.hooks.athena.AWSAthenaHook contrib.hooks.aws_athena_hook.AWSAthenaHook
aws.hooks.aws_dynamodb.AwsDynamoDBHook contrib.hooks.aws_dynamodb_hook.AwsDynamoDBHook
aws.hooks.base_aws.AwsBaseHook contrib.hooks.aws_hook.AwsHook
aws.hooks.datasync.AWSDataSyncHook contrib.hooks.aws_datasync_hook.AWSDataSyncHook
aws.hooks.emr.EmrHook contrib.hooks.emr_hook.EmrHook
aws.hooks.glue_catalog.AwsGlueCatalogHook contrib.hooks.aws_glue_catalog_hook.AwsGlueCatalogHook
aws.hooks.lambda_function.AwsLambdaHook contrib.hooks.aws_lambda_hook.AwsLambdaHook
aws.hooks.logs.AwsLogsHook contrib.hooks.aws_logs_hook.AwsLogsHook
aws.hooks.s3.S3Hook hooks.S3_hook.S3Hook
aws.hooks.sagemaker.SageMakerHook contrib.hooks.sagemaker_hook.SageMakerHook
aws.hooks.sns.AwsSnsHook contrib.hooks.aws_sns_hook.AwsSnsHook
aws.hooks.sqs.SQSHook contrib.hooks.aws_sqs_hook.SQSHook

Secrets

Moved secrets

Airflow 2.0 protocols: airflow.providers.amazon package Airflow 1.10.* previous location (usually airflow.contrib)
aws.secrets.secrets_manager.SecretsManagerBackend contrib.secrets.aws_secrets_manager.SecretsManagerBackend
aws.secrets.systems_manager.SystemsManagerParameterStoreBackend contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend

Releases

Release 2020.6.24

Commit Committed Subject
12af6a080 2020-06-19 Final cleanup for 2020.6.23rc1 release preparation (#9404)
992a18c84 2020-06-19 Move MySqlToS3Operator to transfers (#9400)
a60f589aa 2020-06-19 Add MySqlToS3Operator (#9054)
c7e5bce57 2020-06-19 Prepare backport release candidate for 2020.6.23rc1 (#9370)
40bf8f28f 2020-06-18 Detect automatically the lack of reference to the guide in the operator descriptions (#9290)
f6bd817a3 2020-06-16 Introduce 'transfers' packages (#9320)
58a8ec0e4 2020-06-16 AWSBatchOperator <> ClientHook relation changed to composition (#9306)
a80cd25e8 2020-06-15 Close/Flush byte stream in s3 hook load_string and load_bytes (#9211)
ffb857403 2020-06-14 Decrypt secrets from SystemsManagerParameterStoreBackend (#9214)
a69b031f2 2020-06-10 Add S3ToRedshift example dag and system test (#8877)
17adcea83 2020-06-02 Fix handling of subprocess error handling in s3_file_transform and gcs (#9106)
357e11e0c 2020-05-29 Add Delete/Create S3 bucket operators (#8895)
1ed171bfb 2020-05-28 Add script_args for S3FileTransformOperator (#9019)
0b0e4f7a4 2020-05-26 Preparing for RC3 relase of backports (#9026)
00642a46d 2020-05-26 Fixed name of 20 remaining wrongly named operators. (#8994)
1d36b0303 2020-05-23 Fix references in docs (#8984)
f946f96da 2020-05-23 Old json boto compat removed from dynamodb_to_s3 operator (#8987)
375d1ca22 2020-05-19 Release candidate 2 for backport packages 2020.05.20 (#8898)
12c5e5d8a 2020-05-17 Prepare release candidate for backport packages (#8891)
f3521fb0e 2020-05-16 Regenerate readme files for backport package release (#8886)
f4edd90a9 2020-05-16 Speed up TestAwsLambdaHook by not actually running a function (#8882)
92585ca4c 2020-05-15 Added automated release notes generation for backport operators (#8807)
85bbab27d 2020-05-15 Add EMR operators howto docs (#8863)
e61b9bb9b 2020-05-13 Add AWS EMR System tests (#8618)
ed3f5131a 2020-05-13 Correctly pass sleep time from AWSAthenaOperator down to the hook. (#8845)
7236862a1 2020-05-12 [AIRFLOW-2310] Enable AWS Glue Job Integration (#6007)
d590e5e76 2020-05-11 Add option to propagate tags in ECSOperator (#8811)
0c3db84c3 2020-05-11 [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor (#7731)
cbebed2b4 2020-05-10 Allow passing backend_kwargs to AWS SSM client (#8802)
c7788a689 2020-05-10 Add imap_attachment_to_s3 example dag and system test (#8669)
ff5b70149 2020-05-07 Add google_api_to_s3_transfer example dags and system tests (#8581)
4421f011e 2020-05-01 Improve template capabilities of EMR job and step operators (#8572)
379a884d6 2020-04-28 fix: aws hook should work without conn id (#8534)
74bc316c5 2020-04-27 [AIRFLOW-4438] Add Gzip compression to S3_hook (#8571)
7ea66a1a9 2020-04-26 Add example DAG for ECSOperator (#8452)
b6434dedf 2020-04-24 [AIRFLOW-7111] Add generate_presigned_url method to S3Hook (#8441)
becedd5af 2020-04-19 Remove unrelated EC2 references in ECSOperator (#8451)
ab1290cb0 2020-04-13 Make launch_type parameter optional (#8248)
87969a350 2020-04-09 [AIRFLOW-6515] Change Log Levels from Info/Warn to Error (#8170)
b46d6c060 2020-04-08 Add support for AWS Secrets Manager as Secrets Backend (#8186)
68d1714f2 2020-04-04 [AIRFLOW-6822] AWS hooks should cache boto3 client (#7541)
8a0240257 2020-03-31 Rename CloudBaseHook to GoogleBaseHook and move it to google.common (#8011)
7239d9a82 2020-03-28 Get Airflow Variables from AWS Systems Manager Parameter Store (#7945)
eb4af4f94 2020-03-28 Make BaseSecretsBackend.build_path generic (#7948)
438da7241 2020-03-28 [AIRFLOW-5825] SageMakerEndpointOperator is not idempotent (#7891)
686d7d50b 2020-03-25 Standardize SecretBackend class names (#7846)
eef87b995 2020-03-23 [AIRFLOW-7105] Unify Secrets Backend method interfaces (#7830)
5648dfbc3 2020-03-23 Add missing call to Super class in 'amazon', 'cloudant & 'databricks' providers (#7827)
a36002412 2020-03-19 [AIRFLOW-5705] Make AwsSsmSecretsBackend consistent with VaultBackend (#7753)
2a54512d7 2020-03-17 [AIRFLOW-5705] Fix bugs in AWS SSM Secrets Backend (#7745)
a8b5fc74d 2020-03-16 [AIRFLOW-4175] S3Hook load_file should support ACL policy paramete (#7733)
e31e9ddd2 2020-03-14 [AIRFLOW-5705] Add secrets backend and support for AWS SSM (#6376)
3bb60afc7 2020-03-13 [AIRFLOW-6975] Base AWSHook AssumeRoleWithSAML (#7619)
c0c5f11ad 2020-03-12 [AIRFLOW-6884] Make SageMakerTrainingOperator idempotent (#7598)
b7cdda1c6 2020-03-10 [AIRFLOW-4438] Add Gzip compression to S3_hook (#7680)
42eef3821 2020-03-07 [AIRFLOW-6877] Add cross-provider dependencies as extras (#7506)
9a94ab246 2020-03-01 [AIRFLOW-6962] Fix compeleted to completed (#7600)
1b38f6d9b 2020-02-29 [AIRFLOW-5908] Add download_file to S3 Hook (#6577)
3ea3e1a2b 2020-02-26 [AIRFLOW-6824] EMRAddStepsOperator problem with multi-step XCom (#7443)
6eaa7e3b1 2020-02-25 [AIRFLOW-5924] Automatically unify bucket name and key in S3Hook (#6574)
3320e432a 2020-02-24 [AIRFLOW-6817] Lazy-load airflow.DAG to keep user-facing API untouched (#7517)
7d0e7122d 2020-02-24 [AIRFLOW-6830] Add Subject/MessageAttributes to SNS hook and operator (#7451)
4d03e33c1 2020-02-22 [AIRFLOW-6817] remove imports from airflow/__init__.py, replaced implicit imports with explicit imports, added entry to UPDATING.MD - squashed/rebased (#7456)
47a922b86 2020-02-21 [AIRFLOW-6854] Fix missing typing_extensions on python 3.8 (#7474)
9cbd7de6d 2020-02-18 [AIRFLOW-6792] Remove _operator/_hook/_sensor in providers package and add tests (#7412)
58c3542ed 2020-02-12 [AIRFLOW-5231] Fix S3Hook.delete_objects method (#7375)
b7aa778b3 2020-02-10 [AIRFLOW-6767] Correct name for default Athena workgroup (#7394)
9282185e6 2020-02-09 [AIRFLOW-6761] Fix WorkGroup param in AWSAthenaHook (#7386)
94fccca97 2020-02-04 [AIRFLOW-XXXX] Add pre-commit check for utf-8 file encoding (#7347)
f3ad5cf61 2020-02-03 [AIRFLOW-4681] Make sensors module pylint compatible (#7309)
88e40c714 2020-02-03 [AIRFLOW-6716] Fix AWS Datasync Example DAG (#7339)
a311d3d82 2020-02-03 [AIRFLOW-6718] Fix more occurrences of utils.dates.days_ago (#7341)
cb766b05b 2020-02-03 [AIRFLOW-XXXX] Fix Static Checks on CI (#7342)
97a429f9d 2020-02-02 [AIRFLOW-6714] Remove magic comments about UTF-8 (#7338)
7527eddc5 2020-02-02 [AIRFLOW-4364] Make all code in airflow/providers/amazon pylint compatible (#7336)
cf141506a 2020-02-02 [AIRFLOW-6708] Set unique logger names (#7330)
63aa3db88 2020-02-02 [AIRFLOW-6258] Add CloudFormation operators to AWS providers (#6824)
af4157fde 2020-02-02 [AIRFLOW-6672] AWS DataSync - better logging of error message (#7288)
373c6aa4a 2020-01-30 [AIRFLOW-6682] Move GCP classes to providers package (#7295)
83c037873 2020-01-30 [AIRFLOW-6674] Move example_dags in accordance with AIP-21 (#7287)
1988a97e8 2020-01-28 [AIRFLOW-6659] Move AWS Transfer operators to providers package (#7274)
ab10443e9 2020-01-28 [AIRFLOW-6424] Added a operator to modify EMR cluster (#7213)
40246132a 2020-01-28 [AIRFLOW-6654] AWS DataSync - bugfix when creating locations (#7270)
82c0e5aff 2020-01-28 [AIRFLOW-6655] Move AWS classes to providers (#7271)
599e4791c 2020-01-18 [AIRFLOW-6541] Use EmrJobFlowSensor for other states (#7146)
c319e81ca 2020-01-17 [AIRFLOW-6572] Move AWS classes to providers.amazon.aws package (#7178)
941a07057 2020-01-15 [AIRFLOW-6570] Add dag tag for all example dag (#7176)
78d8fe694 2020-01-08 [AIRFLOW-6245] Add custom waiters for AWS batch jobs (#6811)
e0b022725 2019-12-28 [AIRFLOW-6319] Add support for AWS Athena workgroups (#6871)
57da45685 2019-12-24 [AIRFLOW-6333] Bump Pylint to 2.4.4 & fix/disable new checks (#6888)
cf647c27e 2019-12-18 [AIRFLOW-6038] AWS DataSync reworked (#6773)
7502cad28 2019-12-17 [AIRFLOW-6206] Move and rename AWS batch operator [AIP-21] (#6764)
c4c635df6 2019-12-17 [AIRFLOW-6083] Adding ability to pass custom configuration to lambda client. (#6678)
4fb498f87 2019-12-09 [AIRFLOW-6072] aws_hook: Outbound http proxy setting and other enhancements (#6686)
a1e2f8635 2019-11-20 [AIRFLOW-6021] Replace list literal with list constructor (#6617)
baae14084 2019-11-19 [AIRFLOW-5781] AIP-21 Migrate AWS Kinesis to /providers/amazon/aws (#6588)
504cfbac1 2019-11-18 [AIRFLOW-5783] AIP-21 Move aws redshift into providers structure (#6539)
992f0e3ac 2019-11-12 AIRFLOW-5824: AWS DataSync Hook and Operators added (#6512)
c015eb2f6 2019-11-10 [AIRFLOW-5786] Migrate AWS SNS to /providers/amazon/aws (#6502)
3d76fb4bf 2019-11-09 [AIRFLOW-5782] Migrate AWS Lambda to /providers/amazon/aws [AIP-21] (#6518)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for apache-airflow-backport-providers-amazon, version 2020.6.24
Filename, size File type Python version Upload date Hashes
Filename, size apache_airflow_backport_providers_amazon-2020.6.24-py3-none-any.whl (214.7 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size apache-airflow-backport-providers-amazon-2020.6.24.tar.gz (138.4 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page