Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

Takes a deployment in your kubernetes cluster and turns its pod template into a KubernetesPodOperator object.

Project description

Airflow KubernetesPodOperatorFromDeployment

PyPI version PyPI license PyPI pyversions

Or airkupofrod for short, is a tiny package which does one thing - takes a deployment in your kubernetes cluster and turns allows you to use its pod template as a KubernetesPodOperator object. It does this by providing the KubernetesPodOperatorFromDeployment operator.

airkupofrod supports 1.10.9<=airflow<2

Installation and usage

Ensure your airflow image has the python package airkupofrod installed

pip install airkupofrod

Then in your dags:

from airkupofrod.operator import KubernetesPodOperatorFromDeployment

my_kupofrod_task = KubernetesPodOperatorFromDeployment(
    deployment_labels={"app": "my-app"}, # deployment labels to lookup by
    deployment_fields={"metadata.name": "my-app-deploy-template"}, # deployment fields to lookup by
    deployment_namespace="some-ns", # where the deployment lives
    namespace="default", # where the pod will be deployed
    task_id="my-kupofrod-task", 
    dag=dag,
    in_cluster=True, 
) 

You will also need to make sure that a service account attached to your airflow pods has the a role capable of listing deployments bound to it. See role-binding for an example of this.

This is in addition to the role bindings necessary for the KubernetesPodOperator to work which can be seen in the airflow helm chart

Developing

Skaffold is used to test and develop inside kubernetes.

After ensuring you have:

  • Skaffold
  • Helm
  • Some type of k8s cluster available

Run:

skaffold dev --force=false --cleanup=false --status-check=false --port-forward

Then navigate to http://localhost:8080 and enable and trigger a run of the test deployments dag.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for airkupofrod, version 0.7.0
Filename, size File type Python version Upload date Hashes
Filename, size airkupofrod-0.7.0-py3-none-any.whl (11.3 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size airkupofrod-0.7.0.tar.gz (25.1 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page