rudderstack-airflow-provider 1.0.1

airflow provider for rudderstack

The Customer Data Platform for Developers

Website · Documentation · Slack Community

---

RudderStack Airflow Provider

The RudderStack Airflow Provider lets you schedule and trigger your Warehouse Actions syncs from outside RudderStack and integrate them with your existing Airflow workflows.

Installation

pip install rudderstack-airflow-provider

Usage

A simple DAG for triggering syncs for a RudderStack source:

with DAG(
    'rudderstack-sample',
    default_args=default_args,
    description='A simple tutorial DAG',
    schedule_interval=timedelta(days=1),
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=['rs']
) as dag:
    rs_operator = RudderstackOperator(
        source_id='<source-id>',
        task_id='<any-task-id>',
        connection_id='rudderstack_conn'
    )

For the complete code, refer to this example.

Operator Parameters

Parameter	Description	Type	Default
source_id	Valid RudderStack source ID	String	None
task_id	A unique task ID within a DAG	String	None
wait_for_completion	If True, the task will wait for sync to complete.	Boolean	False
connection_id	The Airflow connection to use for connecting to the Rudderstack API.	String	rudderstack_default

The RudderStack operator also supports all the parameters supported by the Airflow base operator.

For details on how to run the DAG in Airflow, refer to the documentation.

Contribute

We would love to see you contribute to this project. Get more information on how to contribute here.

License

The RudderStack Airflow Provider is released under the MIT License.

Contact Us

For more information or queries on this feature, you can contact us or start a conversation in our Slack community.