Kedro-Airflow makes it easy to deploy Kedro projects to Airflow
Apache Airflow is a tool for orchestrating complex workflows and data processing pipelines. The Kedro-Airflow plugin can be used for:
- Rapid pipeline creation in the prototyping phase. You can write Python functions in Kedro without worrying about schedulers, daemons, services or having to recreate the Airflow DAG file.
- Automatic dependency resolution in Kedro. This allows you to bypass Airflow's need to specify the order of your tasks.
- Distributing Kedro tasks across many workers. You can also enable monitoring and scheduling of the tasks' runtimes.
kedro-airflow is a Python plugin. To install it:
pip install kedro-airflow
You can use
kedro-airflow to deploy a Kedro pipeline as an Airflow DAG by following these steps:
Step 1: Generate the DAG file
At the root directory of the Kedro project, run:
kedro airflow create
This command will generate an Airflow DAG file located in the
airflow_dags/ directory in your project.
You can pass a
--pipeline flag to generate the DAG file for a specific Kedro pipeline and an
--env flag to generate the DAG file for a specific Kedro environment.
Step 2: Copy the DAG file to the Airflow DAGs folder.
For more information about the DAGs folder, please visit Airflow documentation.
Step 3: Package and install the Kedro pipeline in the Airflow executor's environment
After generating and deploying the DAG file, you will then need to package and install the Kedro pipeline into the Airflow executor's environment. Please visit the guide to deploy Kedro as a Python package for more details.
What if my DAG file is in a different directory to my project folder?
By default the generated DAG file is configured to live in the same directory as your project as per this template. If your DAG file is located in a different directory to your project, you will need to tweak this manually after running the
kedro airflow create command.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for kedro_airflow-0.5.0-py3-none-any.whl