Skip to main content

Kedro plugin with Airflow on Kubernetes support

Project description

Kedro Airflow K8S Plugin

Python Version License SemVer PyPI version Downloads

Maintainability Test Coverage Documentation Status

About

The main purpose of this plugin is to enable running kedro pipeline with Airflow on Kubernetes Cluster. In difference to kedro-airflow this plugin does not require additional libraries installed in airflow runtime, it uses K8S infrastructure instead. It supports translation from Kedro pipeline DSL to airflow (python API) and generation of airflow DAGs.

The plugin can be used together with kedro-docker to simplify preparation of docker image for pipeline execution.

Documentation

For detailed documentation refer to https://kedro-airflow-k8s.readthedocs.io/

Usage guide

Usage: kedro airflow-k8s [OPTIONS] COMMAND [ARGS]...
 
Options:
  -e, --env TEXT  Environment to use.
  -p, --pipeline TEXT  Pipeline name to pick.
  -h, --help      Show this message and exit.

Commands:
  compile          Create an Airflow DAG for a project
  init             Initializes configuration for the plugin
  list-pipelines   List pipelines generated by this plugin
  run-once         Uploads pipeline to Airflow and runs once
  schedule         Uploads pipeline to Airflow with given schedule
  ui               Open Apache Airflow UI in new browser tab
  upload-pipeline  Uploads pipeline to Airflow DAG location

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kedro-airflow-k8s-0.8.2.tar.gz (34.4 kB view details)

Uploaded Source

File details

Details for the file kedro-airflow-k8s-0.8.2.tar.gz.

File metadata

  • Download URL: kedro-airflow-k8s-0.8.2.tar.gz
  • Upload date:
  • Size: 34.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.18

File hashes

Hashes for kedro-airflow-k8s-0.8.2.tar.gz
Algorithm Hash digest
SHA256 7e1b1563cc111b71721fb96002141b84cffd7aeaccd680cd5b1ddb4fa8d8f70f
MD5 1df899e21d0d855d1fda67265f4cd37b
BLAKE2b-256 8d8fefc44f273f6333c63759d513ce013a7171cc5611abc5f71095af84aecb54

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page