Hightouch Provider for Airflow
Project description
Apache Airflow Provider for Hightouch
Provides an Airflow Operator and Hook for Hightouch. This allows the user to initiate a run for a sync from Airflow.
Installation
Pre-requisites: An environment running apache-airflow
>= 1.10, including >= 2.
pip install airflow-provider-hightouch
Configuration
In the Airflow Connections UI, create a new connection for Hightouch.
Conn ID
:hightouch_default
Conn Type
:HTTP
Host
:https://api.hightouch.com
Password
: enter the API key for your workspace. You can generate an API key from your Workspace Settings
The Operator uses the hightouch_default
connection id by default, but
if needed, you can create additional Airflow Connections and reference them
in the operator
Modules
HightouchTriggerSyncOperator
Starts a Hightouch Sync Run. Requires the sync_id
or the sync_slug
for the sync you wish to
run.
Returns the sync_run_id
of the sync it triggers.
The run is synchronous by default, and the task will be marked complete once the sync is successfully completed.
However, you can request a asynchronous request instead by passing synchronous=False
to the operator.
If the API key is not authorized or if the request is invalid the task will fail. If a run is already in progress, a new run will be triggered following the completion of the existing run.
HightouchSyncRunSensor
Monitors a Hightouch Sync Run. Requires the sync_id
and the sync_run_id
of the sync you wish to monitor.
To obtain the sync_run_id
of a sync triggered in Airflow, we recommend using XComs to pass the return value
of HightouchTriggerSyncOperator
.
Examples
Creating a run is as simple as importing the operator and providing it with a sync_id. An example dag is available as well.
from airflow_provider_hightouch.operators.hightouch import HightouchTriggerSyncOperator
with DAG(....) as dag:
...
my_task = HightouchTriggerSyncOperator(task_id="run_my_sync", sync_id="123")
my_other_task = HightouchTriggerSyncOperator(task_id="run_my_sync", sync_slug="my-sync-slug")
Issues
Please submit issues and pull requests in our official repo: https://github.com/hightouchio/airflow-provider-hightouch
We are happy to hear from you, for any feedback please email the authors at pedram@hightouch.io.
Acknowledgements
Special thanks to Fivetran for their provider and Marcos Marx's Airbyte contribution in the core Airflow repo for doing this before we had to so we could generously learn from their hard work.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file airflow-provider-hightouch-4.0.0.tar.gz
.
File metadata
- Download URL: airflow-provider-hightouch-4.0.0.tar.gz
- Upload date:
- Size: 15.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31782c526366605b8d1e5396093c440efc2a2e04577dbda4e4f3d341dda6d57e |
|
MD5 | d9b43b0a9d1a715a5b00c1a66d70323f |
|
BLAKE2b-256 | b809e873bc9deb5cc9d6a6b65ef470b2aa2bcb92c04808a472c404c4a4274003 |
File details
Details for the file airflow_provider_hightouch-4.0.0-py3-none-any.whl
.
File metadata
- Download URL: airflow_provider_hightouch-4.0.0-py3-none-any.whl
- Upload date:
- Size: 19.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e83f978ca7b250ddf05dac248938cab02ab65893cbfc460a99ec3e4af478ba92 |
|
MD5 | 54306f1442215b761a90fb1ba8861735 |
|
BLAKE2b-256 | 8e22812369277db810cf7af02d985185fde426d5082c6d4b26f7ef0ecb428317 |