Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer
Project description
Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer.
Installation
Install and update using pip:
pip install astronomer-providers
Example Usage
This repo is structured same as the Apache Airflow’s source code, so for example if you want to import Async operators, you can import it as follows:
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync as S3KeySensor
waiting_for_s3_key = S3KeySensor(
task_id="waiting_for_s3_key",
bucket_key="sample_key.txt",
wildcard_match=False,
bucket_name="sample-bucket",
)
Example DAGs for each provider is within the respective provider’s folder. For example, the Kubernetes provider’s DAGs are within the astronomer/providers/cncf/kubernetes/example_dags folder.
Principle
We will only create Async operators for the “sync-version” of operators that do some level of polling (take more than a few seconds to complete).
For example, we won’t create an async Operator for a BigQueryCreateEmptyTableOperator but will create one for BigQueryInsertJobOperator that actually runs queries and can take hours in the worst case for task completion.
Changelog
We follow Semantic Versioning for releases. Check CHANGELOG.rst for the latest changes.
Contributing Guide
All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.
A detailed overview on how to contribute can be found in the Contributing Guide.
As contributors and maintainers to this project, you are expected to abide by the Contributor Code of Conduct.
License
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
File details
Details for the file astronomer_providers-1.0.0-py3-none-any.whl
.
File metadata
- Download URL: astronomer_providers-1.0.0-py3-none-any.whl
- Upload date:
- Size: 72.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/29.0 requests/2.26.0 requests-toolbelt/0.9.1 urllib3/1.26.7 tqdm/4.50.2 importlib-metadata/4.11.1 keyring/21.4.0 rfc3986/1.5.0 colorama/0.4.4 CPython/3.8.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b29be9f47126cb7759c2a13abbede4b11fc402a2c21caa2a83229ccf4828b791 |
|
MD5 | 55d2d8f18763280f7e3f245d5490e722 |
|
BLAKE2b-256 | 6261ded3d626a5beaa3106e2b2f38d4b4086faec64213b4e1b6518fc319116fb |