Apache airflow provider for managing Reverse ETL syncs and Profiles runs in RudderStack.
Project description
The Customer Data Platform for Developers
RudderStack Airflow Provider
The RudderStack Airflow Provider lets you programmatically schedule and trigger your Reverse ETL syncs and Profiles runs outside RudderStack and integrate them with your existing Airflow workflows.
Installation
pip install rudderstack-airflow-provider
Usage
RudderstackRETLOperator
[!NOTE]
Use RudderstackRETLOperator for reverse ETL connections
A simple DAG for triggering syncs for a RudderStack Reverse ETL source:
with DAG(
"rudderstack-retl-sample",
default_args=default_args,
description="A simple tutorial DAG for reverse etl",
schedule_interval=timedelta(days=1),
start_date=datetime(2021, 1, 1),
catchup=False,
tags=["rs-retl"],
) as dag:
# retl_connection_id, sync_type are template fields
rs_operator = RudderstackRETLOperator(
retl_connection_id="connection_id",
task_id="<replace task id>",
connection_id="<rudderstack api airflow connection id>"
)
For the complete code, refer to this example.
Mandatatory parameters for RudderstackRETLOperator:
- retl_connection_id: This is the connection id for the sync job.
- connection_id: The Airflow connection to use for connecting to the Rudderstack API. Default value is
rudderstack_default
.
RudderstackRETLOperator exposes other configurable parameters as well. Mostly default values for them would be recommended.
- request_max_retries: The maximum number of times requests to the RudderStack API should be retried before failng.
- request_retry_delay: Time (in seconds) to wait between each request retry.
- request_timeout: Time (in seconds) after which the requests to RudderStack are declared timed out.
- poll_interval: Time (in seconds) for polling status of triggered job.
- poll_timeout: Time (in seconds) after which the polling for a triggered job is declared timed out.
- wait_for_completion: Boolean if execution run should poll and wait till completion of sync. Default value is True.
- sync_type: Type of sync to trigger
incremental
orfull
. Default is None as RudderStack will be deteriming sync type.
RudderstackProfilesOperator
RudderstackProfilesOperator can be used to trigger profiles run. A simple DAG for triggering profile runs for a profiles project.
with DAG(
"rudderstack-profiles-sample",
default_args=default_args,
description="A simple tutorial DAG for profiles run.",
schedule_interval=timedelta(days=1),
start_date=datetime(2021, 1, 1),
catchup=False,
tags=["rs-profiles"],
) as dag:
# profile_id is template field
rs_operator = RudderstackProfilesOperator(
profile_id="<profile_id>",
task_id="<replace task id>",
connection_id="<rudderstack api connection id>",
)
Mandatatory parameters for RudderstackProfilesOperator:
- profile_id: This is the profiles id for the profiles project to run.
- connection_id: The Airflow connection to use for connecting to the Rudderstack API. Default value is
rudderstack_default
.
RudderstackProfilesOperator exposes other configurable parameters as well. Mostly default values for them would be recommended.
- request_max_retries: The maximum number of times requests to the RudderStack API should be retried before failng.
- request_retry_delay: Time (in seconds) to wait between each request retry.
- request_timeout: Time (in seconds) after which the requests to RudderStack are declared timed out.
- poll_interval: Time (in seconds) for polling status of triggered job.
- poll_timeout: Time (in seconds) after which the polling for a triggered job is declared timed out.
- wait_for_completion: Boolean if execution run should poll and wait till completion of sync. Default value is True.
- parameters: Additional parameters to pass to the profiles run command, as supported by the API endpoint. Default value is
None
.
Contribute
We would love to see you contribute to this project. Get more information on how to contribute here.
License
The RudderStack Airflow Provider is released under the MIT License.
Contact Us
For more information or queries on this feature, you can contact us or start a conversation in our Slack community.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file rudderstack_airflow_provider-2.1.0.tar.gz
.
File metadata
- Download URL: rudderstack_airflow_provider-2.1.0.tar.gz
- Upload date:
- Size: 9.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.1 CPython/3.12.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0ecd554432a5ff855a5cbbd7f0cf8cf421fcc47a9cdae9856a236111b027166f |
|
MD5 | 0b44476d97160c39ca87d000e9fa8587 |
|
BLAKE2b-256 | accc64dee11760325f572e2feb39df9d400ddcd1c6edcea9d93a6df80b372d25 |
File details
Details for the file rudderstack_airflow_provider-2.1.0-py3-none-any.whl
.
File metadata
- Download URL: rudderstack_airflow_provider-2.1.0-py3-none-any.whl
- Upload date:
- Size: 10.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.1 CPython/3.12.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 94d3b81e1f6f7e23bb9755ae30fec91bd1c0c88d04a9dedf15e953fbdb5ec6e5 |
|
MD5 | bfa51caaf21f0e97351b6745f7cbedc0 |
|
BLAKE2b-256 | 39efb23ebd0378bb9cbf362a9664f8c9fef3448ba8aec31a15ce8b6b6bdeb67b |