Skip to main content

Astronomer Providers

PyPI Version PyPI - Python Version PyPI - License Code style: black CodeCov Documentation Status Security: bandit

Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer.

Installation

Install and update using pip:

pip install astronomer-providers

This only installs dependencies for core provider. To install all dependencies, run:

pip install 'astronomer-providers[all]'

To only install the dependencies for a specific provider, specify the integration name as extra argument, example to install Kubernetes provider dependencies, run:

pip install 'astronomer-providers[cncf.kubernetes]'

Extras

Extra Name

Installation Command

Dependencies

all

pip install 'astronomer-providers[all]'

All

amazon

pip install 'astronomer-providers[amazon]'

Amazon

apache.hive

pip install 'astronomer-providers[apache.hive]'

Apache Hive

apache.livy

pip install 'astronomer-providers[apache.livy]'

Apache Livy

cncf.kubernetes

pip install 'astronomer-providers[cncf.kubernetes]'

Cncf Kubernetes

databricks

pip install 'astronomer-providers[databricks]'

Databricks

dbt.cloud

pip install 'astronomer-providers[dbt.cloud]'

Dbt Cloud

google

pip install 'astronomer-providers[google]'

Google

http

pip install 'astronomer-providers[http]'

Http

microsoft.azure

pip install 'astronomer-providers[microsoft.azure]'

Microsoft Azure

openlineage

pip install 'astronomer-providers[openlineage]'

Openlineage

sftp

pip install 'astronomer-providers[sftp]'

Sftp

snowflake

pip install 'astronomer-providers[snowflake]'

Snowflake

Example Usage

This repo is structured same as the Apache Airflow’s source code, so for example if you want to import Async operators, you can import it as follows:

from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync as S3KeySensor

waiting_for_s3_key = S3KeySensor(
    task_id="waiting_for_s3_key",
    bucket_key="sample_key.txt",
    wildcard_match=False,
    bucket_name="sample-bucket",
)

Example DAGs for each provider is within the respective provider’s folder. For example, the Kubernetes provider’s DAGs are within the astronomer/providers/cncf/kubernetes/example_dags folder.

Principle

We will only create Async operators for the “sync-version” of operators that do some level of polling (take more than a few seconds to complete).

For example, we won’t create an async Operator for a BigQueryCreateEmptyTableOperator but will create one for BigQueryInsertJobOperator that actually runs queries and can take hours in the worst case for task completion.

To create async operators, we need to inherit from the corresponding airflow sync operators. If sync version isn’t available, then inherit from airflow BaseOperator.

To create async sensors, we need to inherit from the corresponding sync sensors. If sync version isn’t available, then inherit from airflow BaseSensorOperator.

Changelog

We follow Semantic Versioning for releases. Check CHANGELOG.rst for the latest changes.

Contributing Guide

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

A detailed overview on how to contribute can be found in the Contributing Guide.

As contributors and maintainers to this project, you are expected to abide by the Contributor Code of Conduct.

Goals for the project

  • Our focus is on the speed of iteration and development in this stage of the project and so we want to be able to quickly iterate with our community members and customers and cut releases as necessary

  • Airflow Providers are separate packages from the core apache-airflow package and we would like to avoid further bloating the Airflow repo

  • We want users and the community to be able to easily track features and the roadmap for individual providers that we develop

  • We would love to see the Airflow community members create, maintain and share their providers to build an Ecosystem of Providers.

Limitations

  • In Airflow sensors have a param mode which can be poke and reschedule. In async sensors, this param has no usage since tasks gets deferred to Triggerer.

License

Apache License 2.0

Metadata

Release files for astronomer-providers 1.17.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for astronomer-providers 1.17.1
File Size Uploaded Provenance Actions
astronomer-providers-1.17.1.tar.gz 179.6 kB Without provenance Details for astronomer-providers-1.17.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for astronomer-providers 1.17.1
File Interpreter ABI Platform Actions
astronomer_providers-1.17.1-py3-none-any.whl Python 3 none any Details for astronomer_providers-1.17.1-py3-none-any.whl

Total release size: 448.9 kB

Release files / astronomer-providers-1.17.1.tar.gz

Download URL astronomer-providers-1.17.1.tar.gz
Size 179.6 kB
Tags Source
SHA-256 checksum
How to use checksums
c11d2af7401420c0bf5374979d8f5fa91a78929ed745cd3c80be61ed795db68c
BLAKE2b-256 checksum
How to use checksums
5c745e70836fb79488bd5fd55082d8b238d3ad3a3a7af909b5fdfe0134fd2272
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.9.17

Release files / astronomer_providers-1.17.1-py3-none-any.whl

Download URL astronomer_providers-1.17.1-py3-none-any.whl
Size 269.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4fa40084cbf24ae793adc7f9dee1b581b4047088102bd00c8dfc517a7be7f38f
BLAKE2b-256 checksum
How to use checksums
1b92ec91ce7b3ed702c555cc82750fb6857120474215a781001a1c1f08f0701c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.9.17

Security for astronomer-providers 1.17.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.17.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page