Skip to main content

Hightouch Provider for Airflow

Project description

Apache Airflow Provider for Hightouch

Provides an Airflow Operator and Hook for Hightouch. This allows the user to initiate a run for a sync from Airflow.

Installation

Pre-requisites: An environment running apache-airflow >= 1.10, including >= 2.

pip install airflow-provider-hightouch

Configuration

In the Airflow Connections UI, create a new connection for Hightouch.

  • Conn ID: hightouch_default
  • Conn Type: HTTP
  • Host: https://api.hightouch.com
  • Password: enter the API key for your workspace. You can generate an API key from your Workspace Settings

The Operator uses the hightouch_default connection id by default, but if needed, you can create additional Airflow Connections and reference them in the operator

Modules

HightouchTriggerSyncOperator

Starts a Hightouch Sync Run. Requires the sync_id or the sync_slug for the sync you wish to run.

Returns the sync_run_id of the sync it triggers.

The run is synchronous by default, and the task will be marked complete once the sync is successfully completed.

However, you can request a asynchronous request instead by passing synchronous=False to the operator.

If the API key is not authorized or if the request is invalid the task will fail. If a run is already in progress, a new run will be triggered following the completion of the existing run.

HightouchSyncRunSensor

Monitors a Hightouch Sync Run. Requires the sync_id and the sync_run_id of the sync you wish to monitor. To obtain the sync_run_id of a sync triggered in Airflow, we recommend using XComs to pass the return value of HightouchTriggerSyncOperator.

Examples

Creating a run is as simple as importing the operator and providing it with a sync_id. An example dag is available as well.

from airflow_provider_hightouch.operators.hightouch import HightouchTriggerSyncOperator

with DAG(....) as dag:
...
    my_task = HightouchTriggerSyncOperator(task_id="run_my_sync", sync_id="123")
    my_other_task = HightouchTriggerSyncOperator(task_id="run_my_sync", sync_slug="my-sync-slug")

Issues

Please submit issues and pull requests in our official repo: https://github.com/hightouchio/airflow-provider-hightouch

We are happy to hear from you, for any feedback please email the authors at pedram@hightouch.io.

Acknowledgements

Special thanks to Fivetran for their provider and Marcos Marx's Airbyte contribution in the core Airflow repo for doing this before we had to so we could generously learn from their hard work.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-provider-hightouch-4.0.0.tar.gz (15.5 kB view details)

Uploaded Source

Built Distribution

airflow_provider_hightouch-4.0.0-py3-none-any.whl (19.5 kB view details)

Uploaded Python 3

File details

Details for the file airflow-provider-hightouch-4.0.0.tar.gz.

File metadata

File hashes

Hashes for airflow-provider-hightouch-4.0.0.tar.gz
Algorithm Hash digest
SHA256 31782c526366605b8d1e5396093c440efc2a2e04577dbda4e4f3d341dda6d57e
MD5 d9b43b0a9d1a715a5b00c1a66d70323f
BLAKE2b-256 b809e873bc9deb5cc9d6a6b65ef470b2aa2bcb92c04808a472c404c4a4274003

See more details on using hashes here.

File details

Details for the file airflow_provider_hightouch-4.0.0-py3-none-any.whl.

File metadata

File hashes

Hashes for airflow_provider_hightouch-4.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 e83f978ca7b250ddf05dac248938cab02ab65893cbfc460a99ec3e4af478ba92
MD5 54306f1442215b761a90fb1ba8861735
BLAKE2b-256 8e22812369277db810cf7af02d985185fde426d5082c6d4b26f7ef0ecb428317

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page