Skip to main content

Airflow plugin to export dag and task based metrics to Prometheus.

Project description

Airflow prometheus exporter

Exposes dag and task based metrics from Airflow to a Prometheus compatible endpoint.

Discussion

You can ask questions in Gitter channel: https://gitter.im/epoch8/airflow-exporter

Screenshots

Compatibility

Note: this version is compatible with Airflow 1.10.3+ only, see #46 for details

For compatibility with previous versions of Airflow use older version: v0.5.4

  • Airflow: airflow1.10.3+
  • Python: python2, python3
  • DataBase: postgresql, mysql

Install

pip install airflow-exporter

That's it. You're done.

Exporting extra labels to Prometheus

It is possible to add extra labels to DAG-related metrics by providing labels dict to DAG params.

Example

dag = DAG(
    'dummy_dag',
    schedule_interval=timedelta(hours=5),
    default_args=default_args,
    catchup=False,
    params={
        'labels': {
            'env': 'test'
        }
    }
)

Label env with value test will be added to all metrics related to dummy_dag:

airflow_dag_status{dag_id="dummy_dag",env="test",owner="owner",status="running"} 12.0

Metrics

Metrics will be available at

http://<your_airflow_host_and_port>/admin/metrics/

airflow_task_status

Labels:

  • dag_id
  • task_id
  • owner
  • status
  • hostname

Value: number of tasks in specific status.

airflow_dag_status

Labels:

  • dag_id
  • owner
  • status

Value: number of dags in specific status.

airflow_dag_run_duration

Labels:

  • dag_id: unique identifier for a given DAG

Value: duration in seconds of the longest DAG Run for given DAG. This metric is not available for DAGs that have already completed.

License

Distributed under the BSD license. See LICENSE for more information.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-exporter-1.3.0.tar.gz (4.3 kB view details)

Uploaded Source

Built Distribution

airflow_exporter-1.3.0-py2.py3-none-any.whl (5.8 kB view details)

Uploaded Python 2Python 3

File details

Details for the file airflow-exporter-1.3.0.tar.gz.

File metadata

  • Download URL: airflow-exporter-1.3.0.tar.gz
  • Upload date:
  • Size: 4.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.6

File hashes

Hashes for airflow-exporter-1.3.0.tar.gz
Algorithm Hash digest
SHA256 8a308b32b8070b2a356f675a2e6dcf8cdcde327be18b16768490e8a81a222a60
MD5 b916440fdeb034b9e2d4f6922f9b3624
BLAKE2b-256 99879bb77d1f0c1e9ae12fabaef79085712625fa1190e6bbfb9064bf1a0fd93d

See more details on using hashes here.

File details

Details for the file airflow_exporter-1.3.0-py2.py3-none-any.whl.

File metadata

  • Download URL: airflow_exporter-1.3.0-py2.py3-none-any.whl
  • Upload date:
  • Size: 5.8 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.6

File hashes

Hashes for airflow_exporter-1.3.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 18d782311e4ce4da8ce3b765a53cab1e21369bf295a6485a738dcb36d8697af1
MD5 e42514fab905e572e503e92d38605468
BLAKE2b-256 84b46b6d367c65b65022b9fe5daf87ead1e1519be6ee2253660c070d34609f87

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page