Skip to main content

Kedro plugin with Airflow on Kubernetes support

Project description

Kedro Airflow K8S Plugin

Python Version License SemVer PyPI version Downloads

Maintainability Test Coverage Documentation Status FOSSA Status

About

The main purpose of this plugin is to enable running kedro pipeline with Airflow on Kubernetes Cluster. In difference to kedro-airflow this plugin does not require additional libraries installed in airflow runtime, it uses K8S infrastructure instead. It supports translation from Kedro pipeline DSL to airflow (python API) and generation of airflow DAGs.

The plugin can be used together with kedro-docker to simplify preparation of docker image for pipeline execution.

Documentation

For detailed documentation refer to https://kedro-airflow-k8s.readthedocs.io/

Usage guide

Usage: kedro airflow-k8s [OPTIONS] COMMAND [ARGS]...
 
Options:
  -e, --env TEXT  Environment to use.
  -p, --pipeline TEXT  Pipeline name to pick.
  -h, --help      Show this message and exit.

Commands:
  compile          Create an Airflow DAG for a project
  init             Initializes configuration for the plugin
  list-pipelines   List pipelines generated by this plugin
  run-once         Uploads pipeline to Airflow and runs once
  schedule         Uploads pipeline to Airflow with given schedule
  ui               Open Apache Airflow UI in new browser tab
  upload-pipeline  Uploads pipeline to Airflow DAG location

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kedro-airflow-k8s-0.5.1.tar.gz (20.6 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page