celery_redis_prometheus

Exports task execution metrics in Prometheus format: how many tasks were started and have completed successfully or with failure, and how many tasks are still in the queues (supported only for broker redis).

Inspired by https://gitlab.com/kalibrr/celery-prometheus

Usage

Start HTTP service

Start the HTTP server like this:

$ bin/celery prometheus --host=127.0.0.1 --port=9691

Configure Prometheus

scrape_configs:
  - job_name: 'celery'
    static_configs:
      - targets: ['localhost:9691']

We export the following metrics:

• celery_tasks_total{state="started|succeeded|failed|retried|retries-exceeded", queue="..."}, counter
• celery_task_queuetime_seconds{queue}, histogram (only if task_send_sent_event is enabled in Celery)
• celery_task_runtime_seconds{queue}, histogram

If you pass --queue-length, the queue lengths are read from the broker on each scrape (NOTE: this only works with redis as the broker), resulting in this additional metric:

• celery_queue_length{queue="..."}, gauge

While the broker cannot be reached, celery_queue_length is left out. So an alert like celery_queue_length > 100 silently stops firing; add one that fires when the metric is missing. With a single exporter per alert selector, absent() does that:

- alert: CeleryQueueLengthMissing
  expr: absent(celery_queue_length{namespace="myapp"})
  for: 5m

absent() only fires if no series matches the selector at all. If the selector covers several exporters (e.g. one per broker), one of them failing goes unnoticed as long as another still reports. Compare against the up series Prometheus records for each scrape target instead, which yields one alert per exporter that is down or cannot read its broker:

- alert: CeleryQueueLengthMissing
  expr: up{job="myapp/celery-metrics"} unless on(namespace, pod) celery_queue_length
  for: 5m

The up selector must match only exporters started with --queue-length, and on(...) must name labels that identify the exporter on both sides.

Run tests

With bin/test wrapper script (or uv run pytest).

celery_redis_prometheus changes

2.0.0 (2026-10-12)

• Give up reading queue lengths on a hanging broker after 3 seconds per connect, handshake and query (so about 9 seconds at worst), to still answer within the default 10 second scrape timeout. Configurable via socket_timeout/socket_connect_timeout in broker_transport_options
• Read queue lengths from the broker on each scrape instead of polling them in a background thread. While the broker cannot be reached, celery_queue_length is left out instead of keeping stale values; see the README on how to alert on that
• Replace --queuelength-interval=x with the flag --queue-length. The old option is rejected, so update the exporter arguments when upgrading

2.0.0 (unreleased)

• Nothing changed yet.

1.5.2 (2026-07-27)

• Remove unused setuptools dependency

1.5.1 (2023-10-27)

• Fix brown-bag release

1.5.0 (2023-10-27)

• Be explicit about unregistering default collectors

1.4.0 (2022-10-28)

• Add queue label to task_queuetime and task_runtime metrics

1.3.0 (2022-09-14)

• Add queue label to celery_tasks_total metric
• Add separate state retries-exceeded (previously used failed)

1.2.0 (2022-06-23)

• Update to celery-5.x

1.1.1 (2020-03-20)

• Fix py2 class syntax issues

1.1.0 (2020-03-20)

• Make py2 compatible.

1.0.0 (2020-03-20)

• Initial release