Skip to main content

Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer

Project description

PyPI Version PyPI - Python Version PyPI - License Code style: black CodeCov Documentation Status Security: bandit

Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer.

Installation

Install and update using pip:

pip install astronomer-providers

This only installs dependencies for core provider. To install all dependencies, run:

pip install 'astronomer-providers[all]'

To only install the dependencies for a specific provider, specify the integration name as extra argument, example to install Kubernetes provider dependencies, run:

pip install 'astronomer-providers[cncf.kubernetes]'

Extras

Extra Name

Installation Command

Dependencies

all

pip install 'astronomer-providers[all]'

All

amazon

pip install 'astronomer-providers[amazon]'

Amazon

apache.hive

pip install 'astronomer-providers[apache.hive]'

Apache Hive

apache.livy

pip install 'astronomer-providers[apache.livy]'

Apache Livy

cncf.kubernetes

pip install 'astronomer-providers[cncf.kubernetes]'

Cncf Kubernetes

databricks

pip install 'astronomer-providers[databricks]'

Databricks

dbt.cloud

pip install 'astronomer-providers[dbt.cloud]'

Dbt Cloud

google

pip install 'astronomer-providers[google]'

Google

http

pip install 'astronomer-providers[http]'

Http

microsoft.azure

pip install 'astronomer-providers[microsoft.azure]'

Microsoft Azure

openlineage

pip install 'astronomer-providers[openlineage]'

Openlineage

sftp

pip install 'astronomer-providers[sftp]'

Sftp

snowflake

pip install 'astronomer-providers[snowflake]'

Snowflake

Example Usage

This repo is structured same as the Apache Airflow’s source code, so for example if you want to import Async operators, you can import it as follows:

from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync as S3KeySensor

waiting_for_s3_key = S3KeySensor(
    task_id="waiting_for_s3_key",
    bucket_key="sample_key.txt",
    wildcard_match=False,
    bucket_name="sample-bucket",
)

Example DAGs for each provider is within the respective provider’s folder. For example, the Kubernetes provider’s DAGs are within the astronomer/providers/cncf/kubernetes/example_dags folder.

Principle

We will only create Async operators for the “sync-version” of operators that do some level of polling (take more than a few seconds to complete).

For example, we won’t create an async Operator for a BigQueryCreateEmptyTableOperator but will create one for BigQueryInsertJobOperator that actually runs queries and can take hours in the worst case for task completion.

To create async operators, we need to inherit from the corresponding airflow sync operators. If sync version isn’t available, then inherit from airflow BaseOperator.

To create async sensors, we need to inherit from the corresponding sync sensors. If sync version isn’t available, then inherit from airflow BaseSensorOperator.

Changelog

We follow Semantic Versioning for releases. Check CHANGELOG.rst for the latest changes.

Contributing Guide

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

A detailed overview on how to contribute can be found in the Contributing Guide.

As contributors and maintainers to this project, you are expected to abide by the Contributor Code of Conduct.

Goals for the project

  • Our focus is on the speed of iteration and development in this stage of the project and so we want to be able to quickly iterate with our community members and customers and cut releases as necessary

  • Airflow Providers are separate packages from the core apache-airflow package and we would like to avoid further bloating the Airflow repo

  • We want users and the community to be able to easily track features and the roadmap for individual providers that we develop

  • We would love to see the Airflow community members create, maintain and share their providers to build an Ecosystem of Providers.

Limitations

  • In Airflow sensors have a param mode which can be poke and reschedule. In async sensors, this param has no usage since tasks gets deferred to Triggerer.

License

Apache License 2.0

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

astronomer-providers-1.15.3.tar.gz (171.0 kB view details)

Uploaded Source

Built Distribution

astronomer_providers-1.15.3-py3-none-any.whl (256.4 kB view details)

Uploaded Python 3

File details

Details for the file astronomer-providers-1.15.3.tar.gz.

File metadata

  • Download URL: astronomer-providers-1.15.3.tar.gz
  • Upload date:
  • Size: 171.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.16

File hashes

Hashes for astronomer-providers-1.15.3.tar.gz
Algorithm Hash digest
SHA256 5724d11a5da2e1d6c377835a0d551c10f930c34cc865c48697f7bb26db053d62
MD5 e6c5fff109c2c1e0f42cbab70f32b459
BLAKE2b-256 3e697429914c8d991d86012e02f1577d3c5cedb3815f439067049f65376cc34a

See more details on using hashes here.

File details

Details for the file astronomer_providers-1.15.3-py3-none-any.whl.

File metadata

File hashes

Hashes for astronomer_providers-1.15.3-py3-none-any.whl
Algorithm Hash digest
SHA256 ba8c9183ae8d612b3a27f365c3c8989937cb7f5195b58872db21750a8aada07e
MD5 bdfcbdccca0d07c4b741a78636be1901
BLAKE2b-256 cda02fe6f4a16327d27d9ed470d1f24a983d1e417ca702bb292ef1a92c319e7b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page