Skip to main content

Run and monitor celery tasks

Project description

Summary

Run, monitor and log celery tasks.

Installation and setup

Declare tasks using celery task or cubicweb-celery cwtasks.

On worker side, install cw-celerytask-helpers.

celeryconfig.py example:

CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = BROKER_URL
CUBICWEB_CELERYTASK_REDIS_URL = CELERY_BROKER_URL
CELERY_IMPORTS = ('cw_celerytask_helpers.helpers', 'module.containing.tasks')

In this configuration example, the cw_celerytask_helpers in CELERY_IMPORTS is required to have logging data (in the task) sent back to the Cubicweb instance via Redis. The CUBICWEB_CELERYTASK_REDIS_URL is the Redis endpoint used for this logging handling mechanism.

Start a worker:

# running cubicweb tasks (celeryconfig.py will be imported from your instance config directory)
celery -A cubicweb_celery -i <CW_INSTANCE_NAME> worker -l info

# running pure celery tasks
celery worker -l info

Task state synchronization requires to run the celery-monitor command:

cubicweb-ctl celery-monitor <instance-name>

Ensure to have the celeryconfig.py loaded for both cubicweb instance and celery worker, enforce by settings with CELERY_CONFIG_MODULE environment variable (it must be an importable python module).

Running tasks

Create a task:

from celery import current_app as app
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)

@app.task(name='hi_there')
def my_task(arg, kw=0):
    logger.info('HI %s %s!', arg, kw)
    return 42

Run a task:

from cubicweb_celerytask.entities import start_async_task

cwtask = start_async_task(cnx, 'hi_there', 'THERE', kw=42)
cnx.commit()

start_async_task() accept task names, task objects or task signatures: http://docs.celeryproject.org/en/latest/userguide/canvas.html#signatures

For instance, to start the above task in a dedicated queue named myqueue:

import celery

start_async_task(cnx, celery.signature('hi_there', args=('THERE',),
                                       kwargs={'kw': 42}, queue='myqueue'))

Testing task based application

In CubicWeb test mode, tasks don’t run automatically, use cubicweb_celerytask.entities.get_tasks() to introspect them and cubicweb_celerytask.entities.run_all_tasks() to run them.

Also, CELERY_ALWAYS_EAGER and CELERY_EAGER_PROPAGATES_EXCEPTIONS are set to True by default.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cubicweb_celerytask-2.0.2.tar.gz (21.3 kB view details)

Uploaded Source

Built Distribution

cubicweb_celerytask-2.0.2-py3-none-any.whl (22.8 kB view details)

Uploaded Python 3

File details

Details for the file cubicweb_celerytask-2.0.2.tar.gz.

File metadata

  • Download URL: cubicweb_celerytask-2.0.2.tar.gz
  • Upload date:
  • Size: 21.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.0.1 CPython/3.13.1

File hashes

Hashes for cubicweb_celerytask-2.0.2.tar.gz
Algorithm Hash digest
SHA256 8c5fe21763648980c8af953123f10f183b9e22e510ae97fdd58d8e0bc9803e0f
MD5 4b201ef378b750eee2739891a6647c5a
BLAKE2b-256 33cdabea21520d0823c7d1e535b81b25ee676881905e42a64ed317d191501788

See more details on using hashes here.

File details

Details for the file cubicweb_celerytask-2.0.2-py3-none-any.whl.

File metadata

File hashes

Hashes for cubicweb_celerytask-2.0.2-py3-none-any.whl
Algorithm Hash digest
SHA256 cb820727da7a663d91867e539bcdd68bc619cbd6221c856b8b13502c18082a99
MD5 87040e76755277d07189a8d545cf7099
BLAKE2b-256 4cf8fb39d77febf1193e79072170c5229e06fd08ec462f6957b237c53b40c788

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page