Celery Diagnostics Observer

Standalone observer-first integration for Celery Diagnostics.

The observer runs beside a Celery system. It connects to broker/event sources, samples safe telemetry, and sends sanitized evidence to Celery Diagnostics. It does not install code into customer web or worker processes. An optional scheduler adapter runs only inside Celery Beat when periodic-fire diagnosis is required.

What This Package Is

• A CLI package that provides celery-diagnostics.
• The primary customer integration path for Celery Diagnostics.
• A passive observer for Redis broker queue depth, Celery task/worker events, optional Celery control inspect snapshots, observer health, transport retry, and local sanitized spool.
• An executor for bounded, read-only diagnostic checks requested by Celery Diagnostics. Checks cover task presence, reservation, scheduling, worker capacity and presence, queue consumers, status-only JSON result records in Redis, and task-specific presence in configured Redis queues.
• An optional app-aware observer when run with -A myproject.celery:app; this loads the Celery app inside the observer process to explain routing, visibility timeout, task_track_started, and beat schedule coverage.
• An optional Celery Beat scheduler adapter that reports bounded schedule inventory, due decisions, and publish failures without task payloads.

What This Package Is Not

• It is not producer, web-process, or worker instrumentation.
• It does not install producer or task instrumentation.
• It does not require changing task definitions.
• It does not require a custom Celery Task base.
• It never transmits or persists task args, kwargs, results, raw tracebacks, frame locals, or task message bodies.
• It must not claim worker topology or zero consumers from Redis queue depth alone.

Install

After the first public release:

python -m pip install --upgrade celery-diagnostics

Local development from this directory:

python -m pip install --upgrade -e ".[dev]"

If the shell cannot find celery-diagnostics, activate the virtual environment where it was installed or use:

python -m celery_diagnostics_observer --help

Quick Start

CD_PROJECT_KEY=cd_xxx \
CELERY_BROKER_URL=redis://YOUR_REDIS_HOST:6379/0 \
celery-diagnostics observe \
  --queues default,emails \
  --ingest-url https://ingest.celerydiagnostics.com

CD_PROJECT_KEY is intentionally environment-only. The CLI does not support a --project-key option so project keys are less likely to appear in shell history or process listings.

Commands

observe

Run the long-lived observer process:

CD_PROJECT_KEY=cd_xxx \
CELERY_BROKER_URL=redis://localhost:6379/0 \
celery-diagnostics observe --queues default

Enhanced observer mode loads your Celery app in the observer process only:

CD_PROJECT_KEY=cd_xxx \
celery-diagnostics observe \
  --mode project-aware \
  -A myproject.celery:app

observe prints a startup coverage summary before the runtime loops start. In dry-run mode, stdout stays machine-readable JSON lines.

Read-only diagnostic checks are enabled by default. Disable them explicitly when the observer must remain passive:

celery-diagnostics observe --no-active-probes

For Redis message-presence checks, the observer scans only configured queues, uses the default Kombu priority layout, and applies a bounded scan limit. It parses message envelopes locally only to match the protocol-v2 task ID; message bodies are immediately discarded and are never transmitted or persisted. A missing task is reported only when all scanned lists were stable and decodable; partial or malformed observations remain inconclusive.

Result-backend status checks are advertised only for a Redis backend with the JSON result serializer. A server-side Redis script returns the status field alone, so the Observer does not fetch the task's result value. Other result backends and serializers remain unsupported rather than silently loading a private result record through AsyncResult.state.

Periodic schedules (optional Celery Beat adapter)

The standalone Observer can follow a periodic task after publication, but it cannot know that Celery Beat should have fired an entry and did not. Run Beat with the package's scheduler wrapper when that distinction matters:

CD_PROJECT_KEY=cd_xxx \
CD_INGEST_URL=https://ingest.celerydiagnostics.com \
celery -A myproject.celery:app beat \
  --scheduler celery_diagnostics_observer.beat:ObserverPersistentScheduler

This remains Celery's PersistentScheduler; the wrapper adds sanitized schedule snapshots and evidence for due and failed publish attempts. It does not collect task args, kwargs, result values, broker URLs, or credentials. Schedules above the snapshot cap are reported as an incomplete inventory, so the Backend will not infer that omitted entries were deleted.

Without the adapter, ordinary task diagnosis continues to work and the Periodic schedules page explicitly reports that Beat evidence is unavailable.

doctor

Explain what Celery Diagnostics can and cannot currently know:

CELERY_BROKER_URL=redis://localhost:6379/0 \
celery-diagnostics doctor

For app-aware coverage:

celery-diagnostics doctor --mode project-aware -A myproject.celery:app

doctor does not require CD_PROJECT_KEY. It reports telemetry coverage, safe claims, blocked claims, and next steps. It is a diagnostic coverage explanation, not a fake health check.

status

Show lightweight local integration status:

celery-diagnostics status

Use doctor when you need detailed telemetry coverage.

Configuration

Every CLI option other than the project key can also be passed as an argument. For example:

celery-diagnostics observe \
  --broker redis://localhost:6379/0 \
  --queues default \
  --policy readable

Privacy Defaults

The observer sanitizes telemetry before it leaves the customer environment.

readable sends operational task, queue, routing, and worker identifiers. local-only sends stable HMAC references plus an authenticated encrypted identity capsule. The customer-managed identity key never leaves the Observer, and both modes collect the same lifecycle evidence.

To identify a local-only run, execute this inside the customer environment:

CD_PROJECT_KEY=cf_xxx \
CD_IDENTITY_KEY='customer-managed-secret' \
celery-diagnostics resolve R-XXXXXXXXXXXX

Default behavior:

• no task args;
• no task kwargs;
• no task results;
• no task payload body;
• no raw tracebacks;
• no frame locals;
• project keys appear only in the Authorization header and are never written to event bodies or the local spool;
• broker and ingest URL credentials are redacted in CLI reports.
• non-loopback ingest endpoints must use HTTPS;
• local spool and replay files are written for the owner only (0600).

Redis queue sampling uses safe queue depth checks. Redis-only evidence can show queue pressure and backlog symptoms, but it cannot prove that no worker is consuming a queue.

Project-aware mode uses an allowlist of app configuration facts. It does not dump arbitrary Celery config, task payloads, broker credentials, or exception messages from failed app imports.

The external Observer cannot directly witness an exception that prevents a producer from reaching the broker. In that case it reports the boundary of its evidence instead of inferring a publish failure from absence alone. Progress events are consumed when they already exist in the Celery event stream; this package does not add task-side progress instrumentation.

The optional Beat adapter observes only the Beat-to-broker publication boundary. It does not broaden access to task payloads or application data.

Local Development

Install editable dependencies:

python -m pip install --upgrade -e ".[dev]"

Run tests:

python -m pytest tests -q

Run syntax checks:

python -m py_compile celery_diagnostics_observer/*.py

Build a wheel:

python -m build

Repository Layout

celery_diagnostics_observer/
  active_probes.py    # bounded on-demand read-only checks
  capabilities.py     # advertised active-check capabilities
  app_context.py      # allowlisted app-aware config extraction
  beat.py             # optional privacy-safe Celery Beat scheduler adapter
  cli.py              # celery-diagnostics command implementation
  config.py           # env and CLI config normalization
  coverage.py         # doctor/status/startup coverage model
  event_receiver.py   # Celery event stream receiver
  redis_sampler.py    # Redis queue depth sampler
  redis_message_probe.py # bounded task-id presence check for Redis
  inspect_sampler.py  # optional Celery control inspect sampler
  sanitizer.py        # privacy-safe event normalization
  transport.py        # HTTP transport and retry behavior
  spool.py            # sanitized local JSONL spool
tests/
  test_*.py           # observer unit tests

Release Notes

The package name is celery-diagnostics; its code imports as celery_diagnostics_observer.

Releases are built in GitHub Actions and published to PyPI with Trusted Publishing. No long-lived PyPI API token is stored in GitHub.

Before creating a GitHub release, update pyproject.toml, version.py, and CHANGELOG.md to the same version, then run:

python -m pytest tests -q
python -m py_compile celery_diagnostics_observer/*.py
python -m build
python -m twine check dist/*

Publish by creating a GitHub release tagged vX.Y.Z. The protected pypi environment must approve the publish job. For the first release, configure a pending PyPI Trusted Publisher for repository celerydiagnostics/CeleryDiagnosticsObserver, workflow release.yml, environment pypi, and project celery-diagnostics.

See CHANGELOG.md for version notes.