A Fivetran provider for Apache Airflow
Project description
Apache Airflow Provider for Fivetran
Provides an Airflow operator, sensor, and hook for Fivetran. This allows you to start Fivetran jobs from Airflow, and monitor a Fivetran sync job for completion before running downstream processes.
Fivetran automates your data pipeline, and Airflow automates your data processing.
Installation
Pre-requisites: An environment running apache-airflow
.
pip install airflow-provider-fivetran
Configuration
In the Airflow user interface, you will need to configure a Connection for Fivetran. Most of the Connection config fields will be left blank. Configure the following fields:
Conn Id
:fivetran
Conn Type
:HTTP
Login
: Fivetran API KeyPassword
: Fivetran API Secret
Fivetran API Key and Secret can be found in the Fivetran Account Settings, under the API Config section. Please see our documentation for more information on Fivetran API Authentication.
The sensor and operator assume the Conn Id
is set to fivetran
, however if you are managing multipe Fivetran accounts, you can set this to anything you like. See the DAG in examples to see how to specify a custom Conn Id
.
Modules
Fivetran Operator
Starts a Fivetran sync job. Note that when a Fivetran sync job is controlled via an Operator, it is no longer run on the schedule as managed by Fivetran. In other words, it is now scheduled only from Airflow.
FivetranOperator
requires that you specify the connector_id
of the sync job to start. You can find connector_id
in the Settings page of the connector you configured in the Fivetran interface.
Import into your DAG via:
from fivetran_provider.operators.fivetran import FivetranOperator
Fivetran Sensor
Monitors a Fivetran sync job for completion. This allows you to trigger downstream processes only when one or more Fivetran sync jobs have completed, ensuring data consistency.
Note, it is possible to monitor a sync that is scheduled and manaaged from Fivetran; in other words, you can use FivetranSensor
without using FivetranOperator
. If used in this way, your DAG will wait until the sync job starts on its Fivetran-controlled schedule and then completes.
FivetranSensor
requires that you specify the connector_id
of the sync job to start. You can find connector_id
in the Settings page of the connector you configured in the Fivetran interface.
Import into your DAG via:
from fivetran_provider.sensors.fivetran import FivetranSensor
Examples
See the examples directory for an example DAG.
Issues
Please submit issues and pull requests in our official repo: https://github.com/fivetran/airflow-provider-fivetran
We are happy to hear from you, for any feedback please email the authors at devrel@fivetran.com.
Acknowledgements
Special thanks to Pete DeJoy, Plinio Guzman, and David Koenitzer of Astronomer.io for their contributions and support in getting this provider off the ground.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airflow-provider-fivetran-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 581db68b2d6d2b761c60afb2bb09ce1d14653b522bf615f2fcd64d967b77a0ee |
|
MD5 | 4dfd08057189324dbc4750e716c8fe5a |
|
BLAKE2b-256 | e09550af64d0e60f69cc322787624704d26417454165e24ab4229d802ed14517 |
Hashes for airflow_provider_fivetran-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8edbc9215bfe55c0fec028b9868515f45f0ea0d2995c1c53f3589d752bb8bd1e |
|
MD5 | c841f2543ce0afbf8dabcfbd91520540 |
|
BLAKE2b-256 | d45293c8e70db8277e566e1171d361cf536de4c3f16c4f34582b36c81e7e7aa1 |