Skip to main content

Apache airflow provider for managing Reverse ETL syncs and Profiles runs in RudderStack.

Project description

The Customer Data Platform for Developers

Website · Slack Community


RudderStack Airflow Provider

The RudderStack Airflow Provider lets you programmatically schedule and trigger your Reverse ETL syncs and Profiles runs outside RudderStack and integrate them with your existing Airflow workflows.

Installation

pip install rudderstack-airflow-provider

Usage

RudderstackRETLOperator

[!NOTE]
Use RudderstackRETLOperator for reverse ETL connections

A simple DAG for triggering syncs for a RudderStack Reverse ETL source:

with DAG(
    "rudderstack-retl-sample",
    default_args=default_args,
    description="A simple tutorial DAG for reverse etl",
    schedule_interval=timedelta(days=1),
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=["rs-retl"],
) as dag:
    # retl_connection_id, sync_type are template fields
    rs_operator = RudderstackRETLOperator(
        retl_connection_id="connection_id",
        task_id="<replace task id>",
        connection_id="<rudderstack api airflow connection id>"
    )

For the complete code, refer to this example.

Mandatatory parameters for RudderstackRETLOperator:

  • retl_connection_id: This is the connection id for the sync job.
  • connection_id: The Airflow connection to use for connecting to the Rudderstack API. Default value is rudderstack_default.

RudderstackRETLOperator exposes other configurable parameters as well. Mostly default values for them would be recommended.

  • request_max_retries: The maximum number of times requests to the RudderStack API should be retried before failng.
  • request_retry_delay: Time (in seconds) to wait between each request retry.
  • request_timeout: Time (in seconds) after which the requests to RudderStack are declared timed out.
  • poll_interval: Time (in seconds) for polling status of triggered job.
  • poll_timeout: Time (in seconds) after which the polling for a triggered job is declared timed out.
  • wait_for_completion: Boolean if execution run should poll and wait till completion of sync. Default value is True.
  • sync_type: Type of sync to trigger incremental or full. Default is None as RudderStack will be deteriming sync type.

RudderstackProfilesOperator

RudderstackProfilesOperator can be used to trigger profiles run. A simple DAG for triggering profile runs for a profiles project.

with DAG(
    "rudderstack-profiles-sample",
    default_args=default_args,
    description="A simple tutorial DAG for profiles run.",
    schedule_interval=timedelta(days=1),
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=["rs-profiles"],
) as dag:
    # profile_id is template field
    rs_operator = RudderstackProfilesOperator(
        profile_id="<profile_id>",
        task_id="<replace task id>",
        connection_id="<rudderstack api connection id>",
    )

Mandatatory parameters for RudderstackProfilesOperator:

  • profile_id: This is the profiles id for the profiles project to run.
  • connection_id: The Airflow connection to use for connecting to the Rudderstack API. Default value is rudderstack_default.

RudderstackProfilesOperator exposes other configurable parameters as well. Mostly default values for them would be recommended.

  • request_max_retries: The maximum number of times requests to the RudderStack API should be retried before failng.
  • request_retry_delay: Time (in seconds) to wait between each request retry.
  • request_timeout: Time (in seconds) after which the requests to RudderStack are declared timed out.
  • poll_interval: Time (in seconds) for polling status of triggered job.
  • poll_timeout: Time (in seconds) after which the polling for a triggered job is declared timed out.
  • wait_for_completion: Boolean if execution run should poll and wait till completion of sync. Default value is True.
  • parameters: Additional parameters to pass to the profiles run command, as supported by the API endpoint. Default value is None.

Contribute

We would love to see you contribute to this project. Get more information on how to contribute here.

License

The RudderStack Airflow Provider is released under the MIT License.

Contact Us

For more information or queries on this feature, you can contact us or start a conversation in our Slack community.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rudderstack_airflow_provider-2.1.0.tar.gz (9.7 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file rudderstack_airflow_provider-2.1.0.tar.gz.

File metadata

File hashes

Hashes for rudderstack_airflow_provider-2.1.0.tar.gz
Algorithm Hash digest
SHA256 0ecd554432a5ff855a5cbbd7f0cf8cf421fcc47a9cdae9856a236111b027166f
MD5 0b44476d97160c39ca87d000e9fa8587
BLAKE2b-256 accc64dee11760325f572e2feb39df9d400ddcd1c6edcea9d93a6df80b372d25

See more details on using hashes here.

File details

Details for the file rudderstack_airflow_provider-2.1.0-py3-none-any.whl.

File metadata

File hashes

Hashes for rudderstack_airflow_provider-2.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 94d3b81e1f6f7e23bb9755ae30fec91bd1c0c88d04a9dedf15e953fbdb5ec6e5
MD5 bfa51caaf21f0e97351b6745f7cbedc0
BLAKE2b-256 39efb23ebd0378bb9cbf362a9664f8c9fef3448ba8aec31a15ce8b6b6bdeb67b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page