Skip to main content

Airflow plugin to export dag and task based metrics to Prometheus.

Project description

Airflow prometheus exporter

Exposes dag and task based metrics from Airflow to a Prometheus compatible endpoint.

Compatibility with Airflow versions

>=3

Versions 1.6.0+ are compatible with Airflow 3.0+

>=2.0, <3

Versions 1.5.* are compatible with Airflow 2.0+, but does not work with 3+

<=1.10.14, >=1.10.3

Version v1.3.2 is compatible

Note: Airflow 1.10.14 with Python 3.8 users

You should install importlib-metadata package in order for plugin to be loaded. See #85 for details.

<1.10.3

Version v0.5.4 is compatible

Install

pip install airflow-exporter

That's it. You're done.

Exporting extra labels to Prometheus

It is possible to add extra labels to DAG-related metrics by providing labels dict to DAG params.

Example

dag = DAG(
    'dummy_dag',
    schedule_interval=timedelta(hours=5),
    default_args=default_args,
    catchup=False,
    params={
        'labels': {
            'env': 'test'
        }
    }
)

Label env with value test will be added to all metrics related to dummy_dag:

airflow_dag_status{dag_id="dummy_dag",env="test",owner="owner",status="running"} 12.0

Metrics

Metrics will be available at

http://<your_airflow_host_and_port>/admin/metrics/

airflow_task_status

Labels:

  • dag_id
  • task_id
  • owner
  • status

Value: number of tasks in a specific status.

airflow_dag_status

Labels:

  • dag_id
  • owner
  • status

Value: number of dags in a specific status.

airflow_dag_run_duration

Labels:

  • dag_id: unique identifier for a given DAG

Value: duration in seconds of the longest DAG Run for given DAG. This metric is not available for DAGs that have already finished.

airflow_dag_last_status

Labels:

  • dag_id
  • owner
  • status

Value: 0 or 1 depending on wherever the current state of each dag_id is status.

License

Distributed under the MIT license. See LICENSE for more information.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow_exporter-1.6.0.tar.gz (11.0 kB view details)

Uploaded Source

Built Distribution

airflow_exporter-1.6.0-py2.py3-none-any.whl (6.6 kB view details)

Uploaded Python 2Python 3

File details

Details for the file airflow_exporter-1.6.0.tar.gz.

File metadata

  • Download URL: airflow_exporter-1.6.0.tar.gz
  • Upload date:
  • Size: 11.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.9.22

File hashes

Hashes for airflow_exporter-1.6.0.tar.gz
Algorithm Hash digest
SHA256 1f1aa4a9803b77c407ab93539dff4e50bb0705d26e77184d94c4548c8aca8437
MD5 a00ff5f000a35c3539da9e988d7075c0
BLAKE2b-256 bd4a62ca8986ad70c19a20a3a3485af2792817bf04f4b5b1d8966d25df125541

See more details on using hashes here.

File details

Details for the file airflow_exporter-1.6.0-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for airflow_exporter-1.6.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 6818b40039e45dce6d253f60de99fd9db1dc4c6e471e0e0b53a4ffd4374046bb
MD5 383c25fdfa6e0f68d4d2f813410e79bb
BLAKE2b-256 0d1c0e703d23f96d18f3cd359351023fb018f43403750720328a7ef8a6e8472f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page