Skip to main content

Prometheus Exporter for Airflow Metrics

Project description

Airflow Prometheus Exporter

Build Status

The Airflow Prometheus Exporter exposes various metrics about the Scheduler, DAGs and Tasks which helps improve the observability of an Airflow cluster.

The exporter is based on this prometheus exporter for Airflow.

Requirements

The plugin has been tested with:

  • Airflow >= 1.10.4
  • Python 3.6+

The scheduler metrics assume that there is a DAG named canary_dag. In our setup, the canary_dag is a DAG which has a tasks which perform very simple actions such as establishing database connections. This DAG is used to test the uptime of the Airflow scheduler itself.

Installation

The exporter can be installed as an Airflow Plugin using:

pip install airflow-prometheus-exporter

This should ideally be installed in your Airflow virtualenv.

Metrics

Metrics will be available at

http://<your_airflow_host_and_port>/admin/metrics/

Task Specific Metrics

airflow_task_status

Number of tasks with a specific status.

All the possible states are listed here.

airflow_task_duration

Duration of successful tasks in seconds.

airflow_task_fail_count

Number of times a particular task has failed.

airflow_xcom_param

value of configurable parameter in xcom table

xcom fields is deserialized as a dictionary and if key is found for a paticular task-id, the value is reported as a guage

Add task / key combinations in config.yaml:

xcom_params:
  -
    task_id: abc
    key: count
  -
    task_id: def
    key: errors

a task_id of 'all' will match against all airflow tasks:

xcom_params:
 -
    task_id: all
    key: count

Dag Specific Metrics

airflow_dag_status

Number of DAGs with a specific status.

All the possible states are listed here

airflow_dag_run_duration

Duration of successful DagRun in seconds.

Scheduler Metrics

airflow_dag_scheduler_delay

Scheduling delay for a DAG Run in seconds. This metric assumes there is a canary_dag.

The scheduling delay is measured as the delay between when a DAG is marked as SCHEDULED and when it actually starts RUNNING.

airflow_task_scheduler_delay

Scheduling delay for a Task in seconds. This metric assumes there is a canary_dag.

airflow_num_queued_tasks

Number of tasks in the QUEUED state at any given instance.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow_prometheus_exporter-1.0.8.tar.gz (6.2 kB view details)

Uploaded Source

Built Distribution

airflow_prometheus_exporter-1.0.8-py3-none-any.whl (8.7 kB view details)

Uploaded Python 3

File details

Details for the file airflow_prometheus_exporter-1.0.8.tar.gz.

File metadata

  • Download URL: airflow_prometheus_exporter-1.0.8.tar.gz
  • Upload date:
  • Size: 6.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.6.5

File hashes

Hashes for airflow_prometheus_exporter-1.0.8.tar.gz
Algorithm Hash digest
SHA256 ac6f41c0c23e00e9f8c12a97330fd91220ddc84485460882f18c0df02d3d696f
MD5 49c4897af3bd928c31eddee0ffd20b94
BLAKE2b-256 2b6aba5031cd8b10f9ed8cdc6915c2ec2366770a74268f7f8af367e412bb9040

See more details on using hashes here.

File details

Details for the file airflow_prometheus_exporter-1.0.8-py3-none-any.whl.

File metadata

  • Download URL: airflow_prometheus_exporter-1.0.8-py3-none-any.whl
  • Upload date:
  • Size: 8.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.6.5

File hashes

Hashes for airflow_prometheus_exporter-1.0.8-py3-none-any.whl
Algorithm Hash digest
SHA256 154f238177a866d03d2c0581e0a6ef939b2032a83db4910f80b384c5d5d8ec82
MD5 b3f4bcaebe20c89b15a71cf03550c34f
BLAKE2b-256 d050ecd869b1e06130cc8abf78f59cd6b58fd43ef65e42a695a59be3d2b0bc47

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page