django-celery-task-view

A Celery task monitoring page for Django that reads task results, durations and per-task logs straight from the Redis result backend — no extra database tables, no event consumers, no separate monitoring process to keep alive.

Because the page reads state Celery has already written (plus two small sibling keys written by best-effort worker hooks), an outage of the page can never lose history: whatever is in Redis is what you see. This is deliberately different from Flower, which consumes the worker event stream — events expire in seconds with no consumer attached, and Flower's state is in memory, so a Flower outage becomes a silent hole. It also differs from django-celery-results, which adds a database table and a second write path.

What you get:

• All Tasks / Failures tabs — every task result currently in Redis, newest first, with status, worker, args/kwargs and duration; running tasks sort to the top.
• A detail modal — full meta, result, traceback, and the task's captured log lines (including a still-running task's log so far).
• Multiple result backends on one page — e.g. one per region or deployment — via a {label: redis_url} setting, each row tagged with its source.

Requirements

• Redis result backend (CELERY_RESULT_BACKEND = 'redis://...').

• For the UI, the following django apps installed (the worker hooks and the reader work without them):

  'django_datatables', 'django_modals', 'django_menus', 'ajax_helpers'
  

  from the PyPI packages

  django-filtered-datatables, django-nested-modals, django-tab-menus, django-ajax-helpers
  

Setup

settings.py:

INSTALLED_APPS = [
    ...,
    'celery_task_view',
]

CELERY_RESULT_EXTENDED = True        # without this the meta has no task name / args / worker
CELERY_TASK_TRACK_STARTED = True     # running tasks appear with status STARTED
CELERY_RESULT_EXPIRES = 60 * 60 * 24 * 7   # how much history the page shows

These are read by the worker when it writes the meta, so restart the celery workers after changing them.

Your project's celery.py (for durations and log capture — the page works without this, but shows no Duration or Log):

import celery_task_view.signals  # noqa: F401

urls.py:

path('celery-tasks/', include('celery_task_view.urls')),

The default views require the access_admin permission (override permission_required on a subclass to change this).

Multiple result backends

CELERY_TASK_VIEW_SOURCES = {
    'UK': 'redis://redis:6379/0',
    'DE': 'redis://redis:6379/1',
    'US': 'redis://redis:6379/2',
}

With more than one entry the table gains a Source column and filter, and the detail modal shows which source the task came from. Without the setting the single source is CELERY_RESULT_BACKEND and no Source column is shown. A source whose Redis is unreachable contributes no rows rather than blanking the page.

If sources run with different TIME_ZONEs and USE_TZ off, celery writes local aware datetimes; the page moves them all onto the viewing project's clock so they sort comparably.

Per-task logs

With celery_task_view.signals imported in the worker, a logging handler on the worker's root logger copies every record emitted while a task is running into a capped Redis list (celery-task-log-<uuid>) beside the task's result, with the same TTL. The worker's normal stdout logging is untouched — the Redis copy is additional. print() output is captured too while celery's default worker_redirect_stdouts is on. Records emitted outside any task (worker startup, beat) are not captured.

The log is written line by line as the task runs, and the detail modal is live: while a task is still running the modal re-polls its own url every 2 seconds (refresh_ms on the modal class) and updates the status, duration and log in place. The poll stops by itself when the task finishes or the modal is closed, and each task's modal polls only its own detail, so switching between task modals never shows another task's log. A task that dies without ever writing a result still leaves the lines it got through.

Settings (optional):

CELERY_TASK_VIEW_CAPTURE_LOGS = True   # False turns the handler off
CELERY_TASK_VIEW_LOG_LEVEL = 'INFO'    # minimum level captured
CELERY_TASK_VIEW_LOG_LINES = 1000      # lines kept per task (oldest trimmed)

Every captured line is one Redis round trip; a task that logs in a tight loop pays for it. LOG_LINES caps memory, not traffic.

Demo project

django_examples/ is a runnable demo:

docker compose up --build
# http://localhost:8031
docker compose exec django python manage.py createsuperuser   # to view the default page

The branded page (/branded-celery-tasks/) is open; the default page (/celery-tasks/) requires a logged-in user with the access_admin permission (a superuser qualifies - the demo routes anonymous visitors through the admin login).

The home page queues demo tasks (quick, slow, failing, chatty, print) against two workers on separate result-backend dbs, so the tasks page shows a Source column and filter. slow_task runs long enough to watch: it sorts to the top in STARTED, and its log grows live in the detail modal while it runs. The demo also mounts a branded copy of the page at /branded-celery-tasks/ (see task_examples/views.py and templates/task_examples/branded_tasks.html).

Tests

The suite lives in celery_task_view/tests/ and runs with Django's test runner from the demo project (it needs a real Redis - the reader is SCAN/MGET against one):

docker compose exec django python manage.py test celery_task_view

Tests use two scratch Redis dbs (8 and 9 by default, CELERY_TASK_VIEW_TEST_REDIS / _2 to override) and refuse to touch anything else; lost-task detection is mocked so no broker broadcasts happen. A few warning lines in the output (unreachable source, unreadable json, swallowed redis errors) are the error paths being exercised, not failures.

Lost tasks

A worker restart kills its in-flight tasks, and no final state is ever written for them - their meta would say STARTED (or PROGRESS) until it expired. So for any task whose stored state says it is underway, the page asks that source's workers what they are actually executing (inspect().active(), cached for 10 seconds) and shows the task as LOST when no worker has it. Lost tasks appear on the Failures tab, keep the log lines they got through, sort where they started rather than at the top, and their modal says why and stops polling. RETRY is exempt - a task waiting out its retry countdown sits in the queue, which inspect cannot see.

The broadcast needs the broker reachable from the web process. By default the source urls are used (broker and result backend sharing a Redis db); set CELERY_TASK_VIEW_BROKERS ({label: broker_url}) when they differ. If the broker cannot be asked, nothing is flagged.

CELERY_TASK_VIEW_DETECT_LOST = True    # False turns the check off
CELERY_TASK_VIEW_INSPECT_TIMEOUT = 1.0 # seconds to wait for worker replies

How it works

• Task results live in Redis as celery-task-meta-<uuid> keys; the page SCANs and MGETs them live on each load. Nothing is copied or stored elsewhere.
• Celery never records a task's start time anywhere durable (TRACK_STARTED's meta is overwritten by the final write), so a task_prerun hook stamps celery-task-start-<uuid>; duration is date_done - start, or now - start while running.
• All three keys share the result's TTL (CELERY_RESULT_EXPIRES), so a task's row, start stamp and log expire together.
• The table is list backed: the whole array (capped at max_records, default 5000) is inlined into the page and DataTables handles sort/search/paging client side.
• Task args and results are untrusted input: row values are escaped in Python, and result / traceback / log render escaped inside <pre> in the modal only.

Branding the page

The views build the tab strip and table into a single HTML string, {{ celery_tasks_content }}, so it can be dropped into your own template. Subclass the base views and set template_name:

from celery_task_view.enhanced_views import CeleryTaskListBaseView, CeleryTaskFailuresBaseView
from celery_task_view.modals import CeleryTaskModalMixin

class MyCeleryTaskList(CeleryTaskListBaseView):
    template_name = 'myapp/celery_tasks.html'

class MyCeleryTaskFailures(CeleryTaskFailuresBaseView):
    template_name = 'myapp/celery_tasks.html'

class MyCeleryTaskModal(MyPermissionModal, CeleryTaskModalMixin):   # or (CeleryTaskModalMixin, Modal)
    pass

The template must include the ajax_helpers/datatables/modals libraries and the page script, then place the content wherever it fits your layout. The block carries no outer margins of its own (only a small gap between the tab strip and the table), so wrap it to suit your page:

{% load ajax_helpers %}
{% lib_include 'ajax_helpers' 'Bootstrap' 'FontAwesome' module='ajax_helpers.includes' %}
{% lib_include 'datatable' module='django_datatables.includes' %}
{% lib_include 'Modals' module='django_modals.includes' %}
{{ ajax_helpers_script }}
...
<div class="mt-3 mx-4">
{{ celery_tasks_content }}
</div>

Register the subclasses with celery_task_urlpatterns so the tab menu and modal (which reverse the standard celery_task_view: URL names) point at your views:

from celery_task_view.urls import celery_task_urlpatterns

urlpatterns = [
    path('celery-tasks/', include((celery_task_urlpatterns(
        list_view=MyCeleryTaskList, failures_view=MyCeleryTaskFailures,
        modal_view=MyCeleryTaskModal), 'celery_task_view'))),
]

Gotchas

• djcelery_email sends emails with ignore_result: True — they never appear on the page (and create no result volume).
• Results written before CELERY_RESULT_EXTENDED was enabled show as (name not recorded).
• If your Redis container has no volume, results (and anything queued) are wiped on every container recreate — give it one.
• The log handler only works with the Redis result backend (it reuses the backend's client); on any other backend it does nothing, silently.