Back-ported airflow.providers.google.* package for Airflow 1.10.*
Project description
Package apache-airflow-backport-providers-google
Release: 2020.5.20
Backport package
This is a backport providers package for google provider. All classes for this provider package
are in airflow.providers.google python package.
Only Python 3.6+ is supported for this backport package.
While Airflow 1.10.* continues to support Python 2.7+ - you need to upgrade python to 3.6+ if you want to use this backport package.
Additional notes
The BigQuery classes are undergoing heavy rewriting. The implementation in Airflow 2.0 will not be fully backwards compatible with the old BigQuery operators. Therefore, the BigQuery classes have been stripped out of the current release of the backport packages. We will add the BigQuery Classes back in one of the future releases of the provider's package.
Installation
You can install this package on top of an existing airflow 1.10.* installation via
pip install apache-airflow-backport-providers-google
Compatibility
For full compatibility and test status of the backport packages check Airflow Backport Package Compatibility
PIP requirements
| PIP package | Version required |
|---|---|
| PyOpenSSL | |
| google-ads | >=4.0.0 |
| google-api-python-client | >=1.6.0, <2.0.0dev |
| google-auth | >=1.0.0, <2.0.0dev |
| google-auth-httplib2 | >=0.0.1 |
| google-cloud-automl | >=0.4.0 |
| google-cloud-bigquery-datatransfer | >=0.4.0 |
| google-cloud-bigtable | >=1.0.0 |
| google-cloud-container | >=0.1.1 |
| google-cloud-datacatalog | >=0.5.0,<0.8 |
| google-cloud-dataproc | >=0.5.0 |
| google-cloud-dlp | >=0.11.0 |
| google-cloud-kms | >=1.2.1 |
| google-cloud-language | >=1.1.1 |
| google-cloud-logging | >=1.14.0 |
| google-cloud-monitoring | >=0.34.0 |
| google-cloud-pubsub | >=1.0.0 |
| google-cloud-redis | >=0.3.0 |
| google-cloud-secret-manager | >=0.2.0 |
| google-cloud-spanner | >=1.10.0 |
| google-cloud-speech | >=0.36.3 |
| google-cloud-storage | >=1.16 |
| google-cloud-tasks | >=1.2.1 |
| google-cloud-texttospeech | >=0.4.0 |
| google-cloud-translate | >=1.5.0 |
| google-cloud-videointelligence | >=1.7.0 |
| google-cloud-vision | >=0.35.2 |
| grpcio-gcp | >=0.2.2 |
| httplib2 | ~=0.15 |
| pandas-gbq |
Cross provider package dependencies
Those are dependencies that might be needed in order to use all the features of the package. You need to install the specified backport providers package in order to use them.
You can install such cross-provider dependencies when installing from PyPI. For example:
pip install apache-airflow-backport-providers-google[amazon]
| Dependent package | Extra |
|---|---|
| apache-airflow-backport-providers-amazon | amazon |
| apache-airflow-backport-providers-apache-cassandra | apache.cassandra |
| apache-airflow-backport-providers-cncf-kubernetes | cncf.kubernetes |
| apache-airflow-backport-providers-facebook | |
| apache-airflow-backport-providers-microsoft-azure | microsoft.azure |
| apache-airflow-backport-providers-microsoft-mssql | microsoft.mssql |
| apache-airflow-backport-providers-mysql | mysql |
| apache-airflow-backport-providers-postgres | postgres |
| apache-airflow-backport-providers-presto | presto |
| apache-airflow-backport-providers-sftp | sftp |
Provider class summary
All classes in Airflow 2.0 are in airflow.providers.google package.
Operators
New operators
Moved operators
Sensors
New sensors
Moved sensors
Hooks
New hooks
Moved hooks
Secrets
Moved secrets
Airflow 2.0 protocols: airflow.providers.google package |
Airflow 1.10.* previous location (usually airflow.contrib) |
|---|---|
| cloud.secrets.secrets_manager.CloudSecretsManagerBackend | contrib.secrets.gcp_secrets_manager.CloudSecretsManagerBackend |
Releases
Release 2020.5.20
| Commit | Committed | Subject |
|---|---|---|
| 00642a46d | 2020-05-26 | Fixed name of 20 remaining wrongly named operators. (#8994) |
| 3994030ea | 2020-05-26 | Refactor BigQuery operators (#8858) |
| cdb3f2545 | 2020-05-26 | All classes in backport providers are now importable in Airflow 1.10 (#8991) |
| 1d36b0303 | 2020-05-23 | Fix references in docs (#8984) |
| cf5cf45e1 | 2020-05-23 | Support YAML input for CloudBuildCreateOperator (#8808) |
| 499493c5c | 2020-05-19 | [AIRFLOW-6586] Improvements to gcs sensor (#7197) |
| 375d1ca22 | 2020-05-19 | Release candidate 2 for backport packages 2020.05.20 (#8898) |
| 841d81664 | 2020-05-19 | Allow setting the pooling time in DLPHook (#8824) |
| 12c5e5d8a | 2020-05-17 | Prepare release candidate for backport packages (#8891) |
| f3521fb0e | 2020-05-16 | Regenerate readme files for backport package release (#8886) |
| 15273f0ea | 2020-05-16 | Check for same task instead of Equality to detect Duplicate Tasks (#8828) |
| 92585ca4c | 2020-05-15 | Added automated release notes generation for backport operators (#8807) |
| e1e833bb2 | 2020-05-13 | Update GoogleBaseHook to not follow 308 and use 60s timeout (#8816) |
| 8b5491971 | 2020-05-12 | Refactor BigQuery hook methods to use python library (#8631) |
| 6911dfe83 | 2020-05-12 | Fix template fields in Google operators (#8840) |
| 4b06fde0f | 2020-05-12 | Fix Flake8 errors (#8841) |
| 1d12c347c | 2020-05-12 | Refactor BigQuery check operators (#8813) |
| 493b685d7 | 2020-05-10 | Add separate example DAGs and system tests for google cloud speech (#8778) |
| 79ef8bed8 | 2020-05-10 | Added Upload Multiple Entity Read Files to specified big query dataset (#8610) |
| 280f1f0c4 | 2020-05-10 | Correctly restore upstream_task_ids when deserializing Operators (#8775) |
| 58aefb23b | 2020-05-08 | Added SDFtoGCSOperator (#8740) |
| 723c52c94 | 2020-05-07 | Add documentation for SpannerDeployInstanceOperator (#8750) |
| 25ee4211b | 2020-05-06 | Support all RuntimeEnvironment parameters in DataflowTemplatedJobStartOperator (#8531) |
| 8d6f1aa4b | 2020-05-05 | Support num_retries field in env var for GCP connection (#8700) |
| 67caae0f2 | 2020-05-04 | Add system test for gcs_to_bigquery (#8556) |
| bc45fa675 | 2020-05-03 | Add system test and docs for Facebook Ads operators (#8503) |
| a28c66f23 | 2020-04-30 | [AIRFLOW-4734] Upsert functionality for PostgresHook.insert_rows() (#8625) |
| 992a24ce4 | 2020-04-28 | Split and improve BigQuery example DAG (#8529) |
| c1fb28230 | 2020-04-28 | Refactor BigQueryHook dataset operations (#8477) |
| e8d0f8fea | 2020-04-26 | Improve idempodency in CloudDataTransferServiceCreateJobOperator (#8430) |
| 37fdfa977 | 2020-04-26 | [AIRFLOW-6281] Create guide for GCS to GCS transfer operators (#8442) |
| 14b22e6ff | 2020-04-25 | Add hook and operator for Google Cloud Life Sciences (#8481) |
| 72ddc94d1 | 2020-04-23 | Pass location using parmamter in Dataflow integration (#8382) |
| 912aa4b42 | 2020-04-23 | Added GoogleDisplayVideo360DownloadLineItemsOperator (#8174) |
| 57c8c0583 | 2020-04-22 | Use python client in BQ hook create_empty_table/dataset and table_exists (#8377) |
| 5d3a7eef3 | 2020-04-20 | Allow multiple extra_packages in Dataflow (#8394) |
| 79c99b1b6 | 2020-04-18 | Added location parameter to BigQueryCheckOperator (#8273) |
| 79d3f33c1 | 2020-04-17 | Clean up temporary files in Dataflow operators (#8313) |
| efcffa323 | 2020-04-16 | Add Dataproc SparkR Example (#8240) |
| b198a1fa9 | 2020-04-15 | Create guide for BigQuery operators (#8276) |
| 2636cc932 | 2020-04-14 | Raise exception when GCP credential doesn't support account impersonation (#8213) |
| eee4ebaee | 2020-04-14 | Added Facebook Ads Operator #7887 (#8008) |
| 8cae07ea1 | 2020-04-14 | fixed typo (#8294) |
| 45c898330 | 2020-04-13 | Less aggressive eager upgrade of requirements (#8267) |
| 1fd9ed384 | 2020-04-13 | Add mypy plugin for decorators. (#8145) |
| 327b0a9f7 | 2020-04-13 | Added GoogleDisplayVideo360UploadLineItemsOperator (#8216) |
| bb5e403a3 | 2020-04-10 | Honor schema type for MySQL to GCS data pre-process (#8090) |
| 87969a350 | 2020-04-09 | [AIRFLOW-6515] Change Log Levels from Info/Warn to Error (#8170) |
| 3fc89f29f | 2020-04-06 | [AIRFLOW-7106] Cloud data fusion integration - Allow to pass args to start pipeline (#7849) |
| 7ef75d239 | 2020-04-03 | [AIRFLOW-7117] Honor self.schema in sql_to_gcs as schema to upload (#8049) |
| ed2bc0057 | 2020-04-02 | Add Google Ads list accounts operator (#8007) |
| 3808a6206 | 2020-04-01 | Unify Google class/package names (#8033) |
| 8a0240257 | 2020-03-31 | Rename CloudBaseHook to GoogleBaseHook and move it to google.common (#8011) |
| 8e8978007 | 2020-03-31 | Add more refactor steps for providers.google (#8010) |
| aae3b8fb2 | 2020-03-31 | Individual package READMEs (#8012) |
| 779023968 | 2020-03-30 | [AIRFLOW-7075] Operators for storing information from GCS into GA (#7743) |
| 49abce521 | 2020-03-30 | Improve system tests for Cloud Build (#8003) |
| 0f19a930d | 2020-03-29 | Remove GKEStartPodOperator when backporting (#7908) |
| 0e1c238b2 | 2020-03-28 | Get Airflow Variables from GCP Secrets Manager (#7946) |
| eb4af4f94 | 2020-03-28 | Make BaseSecretsBackend.build_path generic (#7948) |
| 01f99426f | 2020-03-28 | Add download/upload operators for GCS and Google Sheets (#7866) |
| 892522f8e | 2020-03-26 | Change signature of GSheetsHook methods (#7853) |
| bfd425157 | 2020-03-26 | Improve idempotency in MLEngineHook.create_model (#7811) |
| f9c226343 | 2020-03-26 | Fix CloudSecretsManagerBackend invalid connections_prefix (#7861) |
| e3920f12f | 2020-03-26 | Improve setUp/tearDown in Cloud Firestore system test (#7862) |
| 8ba8a7295 | 2020-03-26 | Improve example DAGs for Cloud Memorystore (#7855) |
| f7d1a437c | 2020-03-26 | Fix CloudMemorystoreCreateInstanceAndImportOperator operator (#7856) |
| beef6c230 | 2020-03-26 | Improve authorization in GCP system tests (#7863) |
| 5f165f3e4 | 2020-03-26 | [AIRFLOW-5801] Get GCP credentials from file instead of JSON blob (#7869) |
| 686d7d50b | 2020-03-25 | Standardize SecretBackend class names (#7846) |
| 1982c3fdc | 2020-03-24 | Run Dataflow for ML Engine summary in venv (#7809) |
| eef87b995 | 2020-03-23 | [AIRFLOW-7105] Unify Secrets Backend method interfaces (#7830) |
| 529db07b2 | 2020-03-23 | Improve Google PubSub hook publish method (#7831) |
| 4bde99f13 | 2020-03-23 | Make airflow/providers pylint compatible (#7802) |
| a001489b5 | 2020-03-23 | Improve example DAG for ML Engine (#7810) |
| 9e5a8e7f8 | 2020-03-23 | Add call to Super class in 'google' providers (#7823) |
| b86bf79bf | 2020-03-23 | Fix typo in GCP credentials_provider's docstring (#7818) |
| 56c013ce9 | 2020-03-23 | Add missing docstring in BigQueryHook.create_empty_table (#7817) |
| 426a79847 | 2020-03-23 | Imrove support for laatest API in MLEngineStartTrainingJobOperator (#7812) |
| cdf1809fc | 2020-03-23 | [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager (#7795) |
| 27dac00e1 | 2020-03-22 | [AIRFLOW-7099] Improve system test for cloud transfer service (#7794) |
| 0daf5d729 | 2020-03-22 | Add ability to specify a maximum modified time for objects in GCSToGCSOperator (#7791) |
| c8088c2bd | 2020-03-20 | [AIRFLOW-7100] Add GoogleAnalyticsGetAdsLinkOperator (#7781) |
| 5106a2931 | 2020-03-20 | [AIRFLOW-6752] Add GoogleAnalyticsRetrieveAdsLinksListOperator (#7748) |
| 759ce2a80 | 2020-03-20 | [AIRFLOW-6978] Add PubSubPullOperator (#7766) |
| 6b9b214e4 | 2020-03-20 | [AIRFLOW-6732] Add GoogleAdsHook and GoogleAdsToGcsOperator (#7692) |
| b11891696 | 2020-03-19 | [AIRFLOW-7069] Fix cloudsql system tests (#7770) |
| ae854cae5 | 2020-03-19 | [AIRFLOW-7082] Remove catch_http_exception decorator in GCP hooks (#7756) |
| 7e1e954d2 | 2020-03-19 | [AIRFLOW-7085] Cache credentials, project_id in GCP Base Hook (#7759) |
| 6e21c139b | 2020-03-19 | [AIRFLOW-XXXX] Fix reference to GCP classes in guides (#7762) |
| ce022a3f7 | 2020-03-19 | [AIRFLOW-XXXX] Add cross-references for operators guide (#7760) |
| 029c84e55 | 2020-03-18 | [AIRFLOW-5421] Add Presto to GCS transfer operator (#7718) |
| 63a3102ed | 2020-03-18 | [AIRFLOW-7064] Add CloudFirestoreExportDatabaseOperator (#7725) |
| 73305c7bd | 2020-03-18 | [AIRFLOW-7081] Remove env variables from GCP guide (#7755) |
| 60fdbf6d9 | 2020-03-18 | [AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator (#7728) |
| de7e934ca | 2020-03-17 | [AIRFLOW-7079] Remove redundant code for storing template_fields (#7750) |
| 0de0347b2 | 2020-03-17 | [AIRFLOW-6855]: Escape project_dataset_table in SQL query in gcs to bq … (#7475) |
| 91557c6f8 | 2020-03-17 | [AIRFLOW-7073] GKEStartPodOperator always use connection credentials (#7738) |
| 51161dbd9 | 2020-03-16 | [AIRFLOW-5664] Store timestamps with microseconds precision (#6354) |
| 2bc020c43 | 2020-03-14 | [AIRFLOW-7055] Verbose logging option for google provider (#7711) |
| c997cab42 | 2020-03-13 | [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator (#7630) |
| 137896f32 | 2020-03-12 | [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Op (#7685) |
| 1f77f943d | 2020-03-10 | [AIRFLOW-6980] Improve system tests and building providers package (#7615) |
| bf9b6b6d7 | 2020-03-09 | [AIRFLOW-5013] Add GCP Data Catalog Hook and operators (#7664) |
| e5130dc9f | 2020-03-09 | [AIRFLOW-2911] Add job cancellation capability to Dataflow service (#7659) |
| faf0df4b9 | 2020-03-09 | [AIRFLOW-XXXX] Fix upsert operator in BQ example DAG (#7666) |
| 42eef3821 | 2020-03-07 | [AIRFLOW-6877] Add cross-provider dependencies as extras (#7506) |
| b5b9795f0 | 2020-03-07 | [AIRFLOW-6973] Make GCSCreateBucketOperator idempotent (fix) (#7624) |
| 6b65038fb | 2020-03-06 | [AIRFLOW-6990] Improve system tests for Google Marketing Platform (#7631) |
| 755fe5224 | 2020-03-05 | [AIRFLOW-6915] Add AI Platform Console Link for MLEngineStartTrainingJobOperator (#7535) |
| cb2f33911 | 2020-03-04 | [AIRFLOW-6973] Make GCSCreateBucketOperator idempotent (#7609) |
| 09fea3ce8 | 2020-03-04 | [AIRFLOW-6977] Fix BigQuery DTS example DAG (#7612) |
| 8230ccc48 | 2020-03-04 | [AIRFLOW-6926] Fix Google Tasks operators return types and idempotency (#7547) |
| 0d1e3088a | 2020-03-04 | [AIRFLOW-6970] Improve GCP Video Intelligence system tests (#7604) |
| ab6bb0012 | 2020-03-03 | [AIRFLOW-6971] Fix return type in CloudSpeechToTextRecognizeSpeechOperator (#7607) |
| 3db4ade3d | 2020-02-29 | [AIRFLOW-6924] Fix Google DLP operators return types (#7546) |
| 008b4bab1 | 2020-02-27 | [AIRFLOW-6730] Use total_seconds instead of seconds (#7363) |
| bb552b2d9 | 2020-02-25 | [AIRFLOW-6908] Lazy load AirflowException (#7528) |
| d1a34246a | 2020-02-25 | [AIRFLOW-6593] Add GCP Stackdriver Alerting Hooks and Operators (#7322) |
| 3320e432a | 2020-02-24 | [AIRFLOW-6817] Lazy-load airflow.DAG to keep user-facing API untouched (#7517) |
| dcf874352 | 2020-02-24 | [AIRFLOW-6894] Prevent db query in example_dags (#7516) |
| 4d03e33c1 | 2020-02-22 | [AIRFLOW-6817] remove imports from airflow/__init__.py, replaced implicit imports with explicit imports, added entry to UPDATING.MD - squashed/rebased (#7456) |
| 35b961637 | 2020-02-21 | [AIRFLOW-4973] Add Cloud Data Fusion Pipeline integration (#7486) |
| aff3a361b | 2020-02-20 | [AIRFLOW-6558] Campaign Manager operators for conversions (#7420) |
| 9cbd7de6d | 2020-02-18 | [AIRFLOW-6792] Remove _operator/_hook/_sensor in providers package and add tests (#7412) |
| 5b199cb86 | 2020-02-17 | [AIRFLOW-XXXX] Typo in example_bigquery DAG (#7429) |
| 2c9345a8e | 2020-02-17 | [AIRFLOW-6759] Added MLEngine operator/hook to cancel MLEngine jobs (#7400) |
| 946bdc23c | 2020-02-16 | [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator (#7126) |
| 2381c820c | 2020-02-13 | [AIRFLOW-6505] Let emoji encoded properly for json.dumps() (#7399) |
| 04c1fefbf | 2020-02-03 | [AIRFLOW-6676] added GCSDeleteBucketOperator (#7307) |
| a0252748f | 2020-02-03 | [AIRFLOW-6717] Remove non-existent field from templated_fields (#7340) |
| 97a429f9d | 2020-02-02 | [AIRFLOW-6714] Remove magic comments about UTF-8 (#7338) |
| 9d8d07557 | 2020-02-03 | [AIRFLOW-6715] Fix Google Cloud DLP Example DAG (#7337) |
| cf141506a | 2020-02-02 | [AIRFLOW-6708] Set unique logger names (#7330) |
| 373c6aa4a | 2020-01-30 | [AIRFLOW-6682] Move GCP classes to providers package (#7295) |
| 83c037873 | 2020-01-30 | [AIRFLOW-6674] Move example_dags in accordance with AIP-21 (#7287) |
| 057f3ae3a | 2020-01-29 | [AIRFLOW-6670][depends on AIRFLOW-6669] Move contrib operators to providers package (#7286) |
| ceea293c1 | 2020-01-28 | [AIRFLOW-6656] Fix AIP-21 moving (#7272) |
| c42a375e7 | 2020-01-27 | [AIRFLOW-6644][AIP-21] Move service classes to providers package (#7265) |
| 059eda05f | 2020-01-21 | [AIRFLOW-6610] Move software classes to providers package (#7231) |
| f4d3e5e54 | 2020-01-13 | [AIRFLOW-6102] [AIP-21] Rename Dataproc operators (#7151) |
| e7bf8ecb4 | 2020-01-13 | [AIRFLOW-6119] [AIP-21] Rename GCS operators, hooks and sensors (#7125) |
| 5b6772cb8 | 2020-01-09 | [AIRFLOW-6125] [AIP-21] Rename S3 operator and SFTP operator (#7112) |
| 4f8592ae8 | 2020-01-08 | [AIRFLOW-6118] [AIP-21] Rename Pubsub operators and hook (#7046) |
| 20299473f | 2020-01-03 | [AIRFLOW-6115] [AIP-21] Rename GCP vision operators (#7020) |
| 18e8cea4e | 2020-01-03 | [AIRFLOW-6428] Fix import path for airflow.utils.dates.days_ago in Example DAGs (#7007) |
| 95087af14 | 2019-12-31 | [AIRFLOW-6110] [AIP-21] Rename natural_language service (#6968) |
| 69629a5a9 | 2019-12-09 | [AIRFLOW-5807] Move SFTP from contrib to providers. (#6464) |
| 25e9047a4 | 2019-12-09 | [AIRFLOW-6193] Do not use asserts in Airflow main code (#6749) |
| ed0a14f32 | 2019-12-09 | [AIRFLOW-6120] Rename GoogleCloudBaseHook (#6734) |
| 2f2f89c14 | 2019-12-01 | [AIRFLOW-6139] Consistent spaces in pylint enable/disable (#6701) |
| 03c870a61 | 2019-11-26 | [AIRFLOW-6010] Remove cyclic imports and pylint hacks (#6601) |
| 5c4cfea8c | 2019-11-15 | [AIRFLOW-5718] Add SFTPToGoogleCloudStorageOperator (#6393) |
| 44a8c37a9 | 2019-11-13 | [AIRFLOW-XXX] Fix the docstring for Dataproc get_job method (#6581) |
| d633d3ac4 | 2019-11-13 | [AIRFLOW-5691] Rewrite Dataproc operators to use python library (#6371) |
| d985c02d9 | 2019-11-05 | [AIRFLOW-XXX] Add How-To-Guide to GCP PubSub (#6497) |
| a296cdabd | 2019-11-04 | [AIRFLOW-5743] Move Google PubSub to providers package (#6476) |
| 470b2a779 | 2019-10-30 | [AIRFLOW-5741] Move Cloud Natural Language to providers (#6421) |
| f2caa451f | 2019-10-27 | [AIRFLOW-5742] Move Google Cloud Vision to providers package (#6424) |
| 16d7accb2 | 2019-10-22 | [AIRFLOW-4971] Add Google Display & Video 360 integration (#6170) |
| 4e661f535 | 2019-10-22 | [AIRFLOW-5379] Add Google Search Ads 360 operators (#6228) |
| 19e32b4e2 | 2019-10-18 | [AIRFLOW-5656] Rename provider to providers module (#6333) |
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for apache-airflow-backport-providers-google-2020.5.20rc3.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 23047f8a15f50ea1b2a52f18cc5f4eb1a8100f3674a1035dc8dc4857df8d4e42 |
|
| MD5 | 67ba17f402a89e3711cbf10b19f90b62 |
|
| BLAKE2b-256 | 5c42d9ec614ff3a28c168674de59e03f755deea1ea17a4277424202a919abccc |
Close
Hashes for apache_airflow_backport_providers_google-2020.5.20rc3-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 927480328ea487a71ed7e49df6b9e3f4df188309af4c46af4164e0a283b9391f |
|
| MD5 | fca7587d4aee4bf92ba219d262580d39 |
|
| BLAKE2b-256 | c968433cefe20ae84c72a48851000cbf7135478305f724431959c5fad2f87543 |
