Skip to main content

A Fivetran provider for Apache Airflow

Project description

Fivetran Provider for Apache Airflow

This package provides an operator, sensor, and hook that integrates Fivetran into Apache Airflow. FivetranOperator allows you to start Fivetran jobs from Airflow and FivetranSensor allows you to monitor a Fivetran sync job for completion before running downstream processes.

Fivetran automates your data pipeline, and Airflow automates your data processing.

Installation

Prerequisites: An environment running apache-airflow.

pip install airflow-provider-fivetran

Configuration

In the Airflow user interface, configure a Connection for Fivetran. Most of the Connection config fields will be left blank. Configure the following fields:

  • Conn Id: fivetran_default
  • Conn Type: Fivetran
  • Fivetran API Key: Your Fivetran API Key
  • Fivetran API Secret: Your Fivetran API Secret

Find the Fivetran API Key and Secret in the Fivetran Account Settings, under the API Config section. See our documentation for more information on Fivetran API Authentication.

The sensor and operator assume the Conn Id is set to fivetran_default, however if you are managing multipe Fivetran accounts, you can set this to anything you like. See the DAG in examples to see how to specify a custom Conn Id.

Modules

Fivetran Operator

FivetranOperator starts a Fivetran sync job. Note that when a Fivetran sync job is controlled via an Operator, it is no longer run on the schedule as managed by Fivetran. In other words, it is now scheduled only from Airflow.

FivetranOperator requires that you specify the connector_id of the sync job to start. You can find connector_id in the Settings page of the connector you configured in the Fivetran dashboard.

Import into your DAG via:

from fivetran_provider.operators.fivetran import FivetranOperator

Fivetran Sensor

FivetranSensor monitors a Fivetran sync job for completion. Monitoring with FivetranSensor allows you to trigger downstream processes only when the Fivetran sync jobs have completed, ensuring data consistency. You can use multiple instances of FivetranSensor to monitor multiple Fivetran connectors.

Note, it is possible to monitor a sync that is scheduled and managed from Fivetran; in other words, you can use FivetranSensor without using FivetranOperator. If used in this way, your DAG will wait until the sync job starts on its Fivetran-controlled schedule and then completes.

FivetranSensor requires that you specify the connector_id of the sync job to start. You can find connector_id in the Settings page of the connector you configured in the Fivetran dashboard.

Import into your DAG via:

from fivetran_provider.sensors.fivetran import FivetranSensor

Examples

See the examples directory for an example DAG.

Issues

Please submit issues and pull requests in our official repo: https://github.com/fivetran/airflow-provider-fivetran

We are happy to hear from you. Please email any feedback to the authors at devrel@fivetran.com.

Acknowledgements

Special thanks to Pete DeJoy, Plinio Guzman, and David Koenitzer of Astronomer.io for their contributions and support in getting this provider off the ground.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-provider-fivetran-1.1.4.tar.gz (18.1 kB view details)

Uploaded Source

Built Distribution

airflow_provider_fivetran-1.1.4-py3-none-any.whl (25.4 kB view details)

Uploaded Python 3

File details

Details for the file airflow-provider-fivetran-1.1.4.tar.gz.

File metadata

File hashes

Hashes for airflow-provider-fivetran-1.1.4.tar.gz
Algorithm Hash digest
SHA256 bdd213b032634b92f82c702ec0d442f844185292bd0c85f90b34998b7cc31320
MD5 44d011e09c383f6dd3dcc8e09644933b
BLAKE2b-256 00f4187ef2c73cb5778e3db716750557db5b5fd7c4d1d7153d38ecaa65609796

See more details on using hashes here.

File details

Details for the file airflow_provider_fivetran-1.1.4-py3-none-any.whl.

File metadata

File hashes

Hashes for airflow_provider_fivetran-1.1.4-py3-none-any.whl
Algorithm Hash digest
SHA256 0096e9520034f06315ebcdacd38a11fa313a50853c2e5ea38bcba930b3a138c6
MD5 0a3527509efe1437460077573b3b16f1
BLAKE2b-256 a9e93abd717234f8404554db35c2e4265e15d15481e02f61bcc1d1f20c9c652c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page