Skip to main content

A package for deploying airflow to kubernetes with included monitoring (grafana and elasticsearch integration)

Project description

https://github.com/zbrookle/avionix_airflow/workflows/CI/badge.svg?branch=master

Why are there so many guides on how to set up Apache Airflow when the code could just be given to you?!

Welcome to Avionix Airflow, an out of the box solution to installing airflow on Kubernetes.

Currently supported platforms are Minikube and AWS EKS (managed nodegroups only), pull requests adding support for GKE and AKS are welcome.

Overview of Avionix Airflow Capabilities

Avionix airflow provides the following out of the box solutions for airflow

  1. Airflow Webserver and Scheduler with configured RBAC for the KubernetesExecutor

  2. Airflow metric collection (statsd -> telegraf -> elasticsearch)

  3. Airflow log collection (stdout -> filebeat -> elasticsearch)

  4. Metric and log visualization (elasticsearch & postgres -> grafana)

Installation

pip install avionix_airflow

Requirements

Configuration

Avionix airflow is configured by using Options objects, there are currently 4 different objects:

  1. AirflowOptions

  2. PostgresOptions

  3. MonitoringOptions

  4. CloudOptions (Specified using LocalOptions or AwsOptions)

  5. RedisOptions (Only for use with CeleryExecutor, *not recommended*)

These options are then passed into the function get_chart_builder, which can be used to retrieve an avionix chart builder object. For instruction on how to use the builder object, see avionix

FAQ

  • How do I change the grafana dashboard?
    • You can change the dashboard by setting the MonitoringOptions role to “Admin”

Documentation

Additional documentation can be found here

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

avionix_airflow-0.2.4.tar.gz (64.1 kB view details)

Uploaded Source

File details

Details for the file avionix_airflow-0.2.4.tar.gz.

File metadata

  • Download URL: avionix_airflow-0.2.4.tar.gz
  • Upload date:
  • Size: 64.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.0 requests/2.24.0 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.9.0

File hashes

Hashes for avionix_airflow-0.2.4.tar.gz
Algorithm Hash digest
SHA256 0c567a95dce86294df5c583cf56b0cbc63af3577020c25b7249e12227ca0315f
MD5 7b6d1876f00d149e1bcd2a877a4b4495
BLAKE2b-256 de4424eca3655008935b49f2e923bf661404879e76bd0595a50dbaf325ce1c30

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page