Skip to main content

Airflow plugin for automatically sending metrics from Airflow to Datadog

Project description

Build Status

airflow-metrics

airflow-metrics is an Airflow plugin for automatically sending metrics from Airflow to Datadog.

Tested For: apache-airflow>=1.10.2, <=1.10.3

Installation

pip install airflow-metrics

Optional

If you want to the metrics from BigQueryOperator and GoogleCloudStorageToBigQueryOperator, then make sure the necessary dependencies are installed.

pip install apache-airflow[gcp_api]

Setup

airflow-metrics will report all metrics to Datadog, so create an airflow connection with your Datadog api key.

airflow connections --add --conn_id datadog_default --conn_type HTTP --conn_extr '{"api_key": "<your api key>"}'

Note: If you skip this step, your airflow installation should still work but no metrics will be reported.

Usage

That's it! airflow-metrics will now begin sending metrics from Airflow to Datadog automatically.

Metrics

airflow-metrics will automatically begin reporting the following metrics

  • airflow.task.state The total number of tasks in a state where the state is stored as a tag.
  • airflow.task.state.bq The current number of big query tasks in a state where the state is stored as a tag.
  • airflow.dag.duration The duration of a DAG in ms.
  • airflow.task.duration The duration of a task in ms.
  • airflow.request.duration The duration of a HTTP request in ms.
  • airflow.request.status.success The current number of HTTP requests with successful status codes (<400)
  • airflow.request.status.failure The current number of HTTP requests with unsuccessful status codes (>=400)
  • airflow.task.upserted.bq The number of rows upserted by a BigQueryOperator.
  • airflow.task.delay.bq The time taken for the big query job from a BigQueryOperator to start in ms.
  • airflow.task.duration.bq The time taken for the big query job from a BigQueryOperator to finish in ms.
  • airflow.task.upserted.gcs_to_bq The number of rows upserted by a GoogleCloudStorageToBigQueryOperator.
  • airflow.task.delay.gcs_to_bq The time taken for the big query from a GoogleCloudStorageToBigQueryOperator to start in ms.
  • airflow.task.duration.gcs_to_bq The time taken for the big query from a GoogleCloudStorageToBigQueryOperator to finish in ms.

Configuration

By default, airflow-metrics will begin extracting metrics from Airflow as you run your DAGs and send them to Datadog. You can opt out of it entirely or opt out of a subset of the metrics by setting these configurations in your airflow.cfg

[airflow_metrics]

airflow_metrics_enabled = True
airflow_metrics_tasks_enabled = True
airflow_metrics_bq_enabled = True
airflow_metrics_gcs_to_bq_enabled = True
airflow_metrics_requests_enabled = True
airflow_metrics_thread_enabled = True`

Limitations

airflow-metrics starts a thread to report some metrics, and is not supported when using sqlite as your database.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Getting Started

Set up your virtual environment for python3 however you like.

pip install -e .
airflow initdb
airflow connections --add --conn_id datadog_default --conn_type HTTP --conn_extr '{"api_key": ""}'

Note: The last step is necessary, otherwise the plugin will not initialize correctly and will not collect metrics. But you are free to add a dummy key for development purposes.

Running Tests

pip install -r requirements-dev.txt
pytest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-metrics-0.1.5.tar.gz (8.0 kB view details)

Uploaded Source

Built Distribution

airflow_metrics-0.1.5-py3-none-any.whl (16.5 kB view details)

Uploaded Python 3

File details

Details for the file airflow-metrics-0.1.5.tar.gz.

File metadata

  • Download URL: airflow-metrics-0.1.5.tar.gz
  • Upload date:
  • Size: 8.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.6.8

File hashes

Hashes for airflow-metrics-0.1.5.tar.gz
Algorithm Hash digest
SHA256 864f034bc2c9881d47c1c2753e673b525e895fac644ae21d1204cf12006badb8
MD5 3df278a5231ff311756daa0418de763c
BLAKE2b-256 3984191fb3d6f42d26e801ca076d1db4e55a0e4d94d9b89e7fe95f83a9a2c4c2

See more details on using hashes here.

File details

Details for the file airflow_metrics-0.1.5-py3-none-any.whl.

File metadata

  • Download URL: airflow_metrics-0.1.5-py3-none-any.whl
  • Upload date:
  • Size: 16.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.6.8

File hashes

Hashes for airflow_metrics-0.1.5-py3-none-any.whl
Algorithm Hash digest
SHA256 13f7f05646ba7a7a3b81abc37bfa3ef1d180202232ee620338de107742ccf270
MD5 673799a3c85b210784b135e9b895ddcc
BLAKE2b-256 eafc58bc930282268fc2f3bc0d65856e8e564c78f53b4a7a3058dfe9465ca5d7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page