Skip to main content

A Fivetran async provider for Apache Airflow

Project description

Fivetran Async Provider for Apache Airflow

This package provides an async operator, sensor and hook that integrates Fivetran into Apache Airflow. FivetranSensorAsync allows you to monitor a Fivetran sync job for completion before running downstream processes. FivetranOperatorAsync submits a Fivetran sync job and polls for its status on the triggerer. Since an async sensor or operator frees up worker slot while polling is happening on the triggerer, they consume less resources when compared to traditional "sync" sensors and operators.

Fivetran automates your data pipeline, and Airflow automates your data processing.

Installation

Prerequisites: An environment running apache-airflow.

pip install airflow-provider-fivetran-async

Configuration

In the Airflow user interface, configure a Connection for Fivetran. Most of the Connection config fields will be left blank. Configure the following fields:

  • Conn Id: fivetran
  • Conn Type: Fivetran
  • Login: Fivetran API Key
  • Password: Fivetran API Secret

Find the Fivetran API Key and Secret in the Fivetran Account Settings, under the API Config section. See our documentation for more information on Fivetran API Authentication.

The sensor assumes the Conn Id is set to fivetran, however if you are managing multiple Fivetran accounts, you can set this to anything you like. See the DAG in examples to see how to specify a custom Conn Id.

Modules

Fivetran Operator Async

FivetranOperatorAsync submits a Fivetran sync job and monitors it on trigger for completion. It requires that you specify the connector_id of the sync job to start. You can find connector_id in the Settings page of the connector you configured in the Fivetran dashboard.

Import into your DAG via:

from fivetran_provider_async.operators import FivetranOperatorAsync

Fivetran Sensor Async

FivetranSensorAsync monitors a Fivetran sync job for completion. Monitoring with FivetranSensorAsync allows you to trigger downstream processes only when the Fivetran sync jobs have completed, ensuring data consistency.

You can use multiple instances of FivetranSensorAsync to monitor multiple Fivetran connectors.

If used in this way,

FivetranSensorAsync requires that you specify the connector_id of the sync job to start. You can find connector_id in the Settings page of the connector you configured in the Fivetran dashboard.

Import into your DAG via:

from fivetran_provider_async.sensors import FivetranSensorAsync

Examples

See the examples directory for an example DAG.

Issues

Please submit issues and pull requests in our official repo: https://github.com/astronomer/airflow-provider-fivetran-async

We are happy to hear from you. Please email any feedback to the authors at humans@astronomer.io.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-provider-fivetran-async-1.0.0.tar.gz (16.8 kB view hashes)

Uploaded Source

Built Distribution

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page