cubicweb-celerytask

Summary

Run, monitor and log celery tasks.

Installation and setup

Declare tasks using celery task or cubicweb-celery cwtasks.

On worker side, install cw-celerytask-helpers.

celeryconfig.py example:

CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = BROKER_URL
CUBICWEB_CELERYTASK_REDIS_URL = CELERY_BROKER_URL
CELERY_IMPORTS = ('cw_celerytask_helpers.helpers', 'module.containing.tasks')

In this configuration example, the cw_celerytask_helpers in CELERY_IMPORTS is required to have logging data (in the task) sent back to the Cubicweb instance via Redis. The CUBICWEB_CELERYTASK_REDIS_URL is the Redis endpoint used for this logging handling mechanism.

Start a worker:

# running cubicweb tasks (celeryconfig.py will be imported from your instance config directory)
celery -A cubicweb_celery -i <CW_INSTANCE_NAME> worker -l info

# running pure celery tasks
celery worker -l info

Task state synchronization requires to run the celery-monitor command:

cubicweb-ctl celery-monitor <instance-name>

Ensure to have the celeryconfig.py loaded for both cubicweb instance and celery worker, enforce by settings with CELERY_CONFIG_MODULE environment variable (it must be an importable python module).

Running tasks

Create a task:

from celery import current_app as app
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)

@app.task(name='hi_there')
def my_task(arg, kw=0):
    logger.info('HI %s %s!', arg, kw)
    return 42

Run a task:

from cubicweb_celerytask.entities import start_async_task

cwtask = start_async_task(cnx, 'hi_there', 'THERE', kw=42)
cnx.commit()

start_async_task() accept task names, task objects or task signatures: http://docs.celeryproject.org/en/latest/userguide/canvas.html#signatures

For instance, to start the above task in a dedicated queue named myqueue:

import celery

start_async_task(cnx, celery.signature('hi_there', args=('THERE',),
                                       kwargs={'kw': 42}, queue='myqueue'))

Testing task based application

In CubicWeb test mode, tasks don’t run automatically, use cubicweb_celerytask.entities.get_tasks() to introspect them and cubicweb_celerytask.entities.run_all_tasks() to run them.

Also, CELERY_ALWAYS_EAGER and CELERY_EAGER_PROPAGATES_EXCEPTIONS are set to True by default.

Release files for cubicweb-celerytask 2.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cubicweb-celerytask 2.0.2
File	Size	Uploaded
cubicweb_celerytask-2.0.2.tar.gz	21.3 kB	Jan 7, 2025	Details

Built distribution (wheel)

Table of built distributions (wheels) for cubicweb-celerytask 2.0.2
File	Interpreter	ABI	Platform	Reset
cubicweb_celerytask-2.0.2-py3-none-any.whl 22.8 kB Jan 7, 2025	Python 3	none	any	Details

Total release size: 44.1 kB

Release files / cubicweb_celerytask-2.0.2.tar.gz

Download URL	cubicweb_celerytask-2.0.2.tar.gz
Size	21.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`8c5fe21763648980c8af953123f10f183b9e22e510ae97fdd58d8e0bc9803e0f`
BLAKE2b-256 checksum How to use checksums	`33cdabea21520d0823c7d1e535b81b25ee676881905e42a64ed317d191501788`
Upload date	Jan 7, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.0.1 CPython/3.13.1`

Release files / cubicweb_celerytask-2.0.2-py3-none-any.whl

Download URL	cubicweb_celerytask-2.0.2-py3-none-any.whl
Size	22.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`cb820727da7a663d91867e539bcdd68bc619cbd6221c856b8b13502c18082a99`
BLAKE2b-256 checksum How to use checksums	`4cf8fb39d77febf1193e79072170c5229e06fd08ec462f6957b237c53b40c788`
Upload date	Jan 7, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.0.1 CPython/3.13.1`