Kedro plugin with Airflow on Kubernetes support
Project description
Kedro Airflow K8S Plugin
About
The main purpose of this plugin is to enable running kedro pipeline with Airflow on Kubernetes Cluster. In difference to kedro-airflow this plugin does not require additional libraries installed in airflow runtime, it uses K8S infrastructure instead. It supports translation from Kedro pipeline DSL to airflow (python API) and generation of airflow DAGs.
The plugin can be used together with kedro-docker
to simplify preparation of docker image for pipeline execution.
Documentation
For detailed documentation refer to https://kedro-airflow-k8s.readthedocs.io/
Usage guide
Usage: kedro airflow-k8s [OPTIONS] COMMAND [ARGS]...
Options:
-e, --env TEXT Environment to use.
-p, --pipeline TEXT Pipeline name to pick.
-h, --help Show this message and exit.
Commands:
compile Create an Airflow DAG for a project
init Initializes configuration for the plugin
list-pipelines List pipelines generated by this plugin
run-once Uploads pipeline to Airflow and runs once
schedule Uploads pipeline to Airflow with given schedule
ui Open Apache Airflow UI in new browser tab
upload-pipeline Uploads pipeline to Airflow DAG location
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file kedro-airflow-k8s-0.8.2.tar.gz
.
File metadata
- Download URL: kedro-airflow-k8s-0.8.2.tar.gz
- Upload date:
- Size: 34.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e1b1563cc111b71721fb96002141b84cffd7aeaccd680cd5b1ddb4fa8d8f70f |
|
MD5 | 1df899e21d0d855d1fda67265f4cd37b |
|
BLAKE2b-256 | 8d8fefc44f273f6333c63759d513ce013a7171cc5611abc5f71095af84aecb54 |