Airflow plugin to export dag and task based metrics to Prometheus.
Project description
Airflow prometheus exporter
Exposes dag and task based metrics from Airflow to a Prometheus compatible endpoint.
Compatibility with Airflow versions
>=2.0
Current version is compatible with Airflow 2.0+
<=1.10.14, >=1.10.3
Version v1.3.2 is compatible
Note: Airflow 1.10.14 with Python 3.8 users
You should install importlib-metadata
package in order for plugin to be loaded. See #85 for details.
<1.10.3
Version v0.5.4 is compatible
Install
pip install airflow-exporter
That's it. You're done.
Exporting extra labels to Prometheus
It is possible to add extra labels to DAG-related metrics by providing labels
dict to DAG params
.
Example
dag = DAG(
'dummy_dag',
schedule_interval=timedelta(hours=5),
default_args=default_args,
catchup=False,
params={
'labels': {
'env': 'test'
}
}
)
Label env
with value test
will be added to all metrics related to dummy_dag
:
airflow_dag_status{dag_id="dummy_dag",env="test",owner="owner",status="running"} 12.0
Metrics
Metrics will be available at
http://<your_airflow_host_and_port>/admin/metrics/
airflow_task_status
Labels:
dag_id
task_id
owner
status
hostname
Value: number of tasks in specific status.
airflow_dag_status
Labels:
dag_id
owner
status
Value: number of dags in specific status.
airflow_dag_run_duration
Labels:
dag_id
: unique identifier for a given DAG
Value: duration in seconds of the longest DAG Run for given DAG. This metric is not available for DAGs that have already completed.
License
Distributed under the BSD license. See LICENSE for more information.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airflow_exporter-1.4.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f4ca2ebd53a2c7775c1534d0651f4fca6fd71c363700e6532c132443f41e8330 |
|
MD5 | 037bc7fa2f502566d91841f01317f82b |
|
BLAKE2b-256 | 3eb202366544f7c63a99bdcfb6315056ef542c1ab5b213f7bc687743b9c072e7 |