Skip to main content

Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer

Project description

PyPI Version PyPI - Python Version PyPI - License Code style: black

Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer.

Installation

Install and update using pip:

pip install astronomer-providers

Example Usage

This repo is structured same as the Apache Airflow’s source code, so for example if you want to import Async operators, you can import it as follows:

from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync as S3KeySensor

waiting_for_s3_key = S3KeySensor(
    task_id="waiting_for_s3_key",
    bucket_key="sample_key.txt",
    wildcard_match=False,
    bucket_name="sample-bucket",
)

Example DAGs for each provider is within the respective provider’s folder. For example, the Kubernetes provider’s DAGs are within the astronomer/providers/cncf/kubernetes/example_dags folder.

Principle

We will only create Async operators for the “sync-version” of operators that do some level of polling (take more than a few seconds to complete).

For example, we won’t create an async Operator for a BigQueryCreateEmptyTableOperator but will create one for BigQueryInsertJobOperator that actually runs queries and can take hours in the worst case for task completion.

Changelog

We follow Semantic Versioning for releases. Check CHANGELOG.rst for the latest changes.

Contributing Guide

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

A detailed overview on how to contribute can be found in the Contributing Guide.

As contributors and maintainers to this project, you are expected to abide by the Contributor Code of Conduct.

License

Apache License 2.0

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

astronomer_providers-1.0.0-py3-none-any.whl (72.4 kB view details)

Uploaded Python 3

File details

Details for the file astronomer_providers-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: astronomer_providers-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 72.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/29.0 requests/2.26.0 requests-toolbelt/0.9.1 urllib3/1.26.7 tqdm/4.50.2 importlib-metadata/4.11.1 keyring/21.4.0 rfc3986/1.5.0 colorama/0.4.4 CPython/3.8.5

File hashes

Hashes for astronomer_providers-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 b29be9f47126cb7759c2a13abbede4b11fc402a2c21caa2a83229ccf4828b791
MD5 55d2d8f18763280f7e3f245d5490e722
BLAKE2b-256 6261ded3d626a5beaa3106e2b2f38d4b4086faec64213b4e1b6518fc319116fb

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page