Skip to main content

Prometheus metrics exporter for Celery

Project description

Celery Exporter

Travis CI Docker Hub

Celery Exporter is a Prometheus metrics exporter for Celery 4, written in python.

Here the list of exposed metrics:

  • celery_tasks_total exposes the number of tasks currently known to the queue labeled by name, state and namespace.
  • celery_tasks_runtime_seconds tracks the number of seconds tasks take until completed as histogram labeled by name and namespace
  • celery_tasks_latency_seconds exposes a histogram of task latency, i.e. the time until tasks are picked up by a worker
  • celery_workers exposes the number of currently probably alive workers

Requirements

Dependencies

The project comes with redis lib already installed, you have to install any other dependency in case you are using other brokers.

Celery app

Celery workers have to be configured to send task-related events: http://docs.celeryproject.org/en/latest/userguide/configuration.html#worker-send-task-events.

Celery Exporter is able to enable events on your workers (see Command Options).


Install and Run

Manual Setup

# Install
$ pip install celery-exporter

# Run
$ celery-exporter

Docker

# Download image
$ docker pull ovalmoney/celery-exporter

# Run it
$ docker run -it --rm ovalmoney/celery-exporter

Command Options

$ celery-exporter --help
Usage: celery-exporter [OPTIONS]

Options:
  -b, --broker-url TEXT      URL to the Celery broker.  [env var:
                             CELERY_EXPORTER_BROKER_URL; default:
                             redis://redis:6379/0]
  -l, --listen-address TEXT  Address the HTTPD should listen on.  [env var:
                             CELERY_EXPORTER_LISTEN_ADDRESS; default:
                             0.0.0.0:9540]
  -m, --max-tasks INTEGER    Tasks cache size.  [env var:
                             CELERY_EXPORTER_MAX_TASKS; default: 10000]
  -n, --namespace TEXT       Namespace for metrics.  [env var:
                             CELERY_EXPORTER_NAMESPACE; default: celery]
  --transport-options TEXT   JSON object with additional options passed to the
                             underlying transport.
  --enable-events            Periodically enable Celery events.
  --tz TEXT                  Timezone used by the celery app.
  --verbose                  Enable verbose logging.
  --version                  Show the version and exit.
  --help                     Show this message and exit.

If you then look at the exposed metrics, you should see something like this:

# HELP celery_workers Number of alive workers
# TYPE celery_workers gauge
celery_workers{namespace="celery"} 1.0
# HELP celery_tasks_total Number of tasks per state
# TYPE celery_tasks_total gauge
celery_tasks_total{name="my_app.tasks.calculate_something",namespace="celery",queue="celery",state="RECEIVED"} 0.0
celery_tasks_total{name="my_app.tasks.calculate_something",namespace="celery",queue="celery",state="PENDING"} 0.0
celery_tasks_total{name="my_app.tasks.calculate_something",namespace="celery",queue="celery",state="STARTED"} 0.0
celery_tasks_total{name="my_app.tasks.calculate_something",namespace="celery",queue="celery",state="RETRY"} 0.0
celery_tasks_total{name="my_app.tasks.calculate_something",namespace="celery",queue="celery",state="FAILURE"} 0.0
celery_tasks_total{name="my_app.tasks.calculate_something",namespace="celery",queue="celery",state="REVOKED"} 0.0
celery_tasks_total{name="my_app.tasks.calculate_something",namespace="celery",queue="celery",state="SUCCESS"} 1.0
celery_tasks_total{name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery",state="RECEIVED"} 3.0
celery_tasks_total{name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery",state="PENDING"} 0.0
celery_tasks_total{name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery",state="STARTED"} 1.0
celery_tasks_total{name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery",state="RETRY"} 2.0
celery_tasks_total{name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery",state="FAILURE"} 1.0
celery_tasks_total{name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery",state="REVOKED"} 0.0
celery_tasks_total{name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery",state="SUCCESS"} 7.0
# HELP celery_tasks_runtime_seconds Task runtime (seconds)
# TYPE celery_tasks_runtime_seconds histogram
celery_tasks_runtime_seconds_bucket{le="0.005",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="0.01",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="0.025",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="0.05",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="0.075",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="0.1",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="0.25",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="0.5",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="0.75",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="1.0",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="2.5",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="5.0",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="7.5",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="10.0",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_bucket{le="+Inf",name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_count{name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 29.0
celery_tasks_runtime_seconds_sum{name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 0.04020289977779612
celery_tasks_runtime_seconds_bucket{le="0.005",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="0.01",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="0.025",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="0.05",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="0.075",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="0.1",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="0.25",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="0.5",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="0.75",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="1.0",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="2.5",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="5.0",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="7.5",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="10.0",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_bucket{le="+Inf",name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_count{name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 2.0
celery_tasks_runtime_seconds_sum{name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 0.00402028997777961
# TYPE celery_tasks_runtime_seconds_created gauge
celery_tasks_runtime_seconds_created{name="my_app.tasks.calculate_something",namespace="celery",queue="celery"} 1.548944949810905e+09
celery_tasks_runtime_seconds_created{name="my_app.tasks.fetch_some_data",namespace="celery",queue="celery"} 1.5489449550243628e+09
# HELP celery_tasks_latency_seconds Seconds between a task is received and started.
# TYPE celery_tasks_latency_seconds histogram
celery_tasks_latency_seconds_bucket{namespace="celery",le="0.005"} 2.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="0.01"} 3.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="0.025"} 4.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="0.05"} 4.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="0.075"} 5.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="0.1"} 5.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="0.25"} 5.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="0.5"} 5.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="0.75"} 5.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="1.0"} 5.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="2.5"} 8.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="5.0"} 11.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="7.5"} 11.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="10.0"} 11.0
celery_tasks_latency_seconds_bucket{namespace="celery",le="+Inf"} 11.0
celery_tasks_latency_seconds_count{namespace="celery"} 11.0
celery_tasks_latency_seconds_sum{namespace="celery"} 16.478713035583496
# TYPE celery_task_latency_created gauge
celery_task_latency_seconds_created{namespace="celery"} 1.5489449475378375e+09

Inspired by @zerok work

https://github.com/zerok/celery-prometheus-exporter

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for celery-exporter, version 1.3.0
Filename, size File type Python version Upload date Hashes
Filename, size celery-exporter-1.3.0.tar.gz (10.8 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page