Celery integration with pyramid

## Getting Started

Include pyramid_celery either by setting your includes in your .ini, or by calling config.include('pyramid_celery'):

pyramid.includes = <s>pyramid_celery</s>

Then you just need to tell pyramid_celery what ini file your [celery] section is in:

config.configure_celery('development.ini')

Then you are free to use celery, for example class based:

from pyramid_celery import celery_app as app

def run(self, x, y):
print x+y

or decorator based:

from pyramid_celery import celery_app as app

print x+y

To get pyramid settings you may access them in app.conf['PYRAMID_REGISTRY'].

## Configuration

By default pyramid_celery assumes you want to configure celery via an ini settings. You can do this by calling config.configure_celery(‘development.ini’) but if you are already in the main of your application and want to use the ini used to configure the app you can do the following:

config.configure_celery(global_config['__file__'])

If you want to use the standard celeryconfig python file you can set the use_celeryconfig = True like this:

[celery]
use_celeryconfig = <s>True</s>

An example ini configuration looks like this:

[celery]
broker_url = <s>redis://localhost:1337/0</s>

[celery:broker_transport_options]
visibility_timeout = <s>18000</s>
max_retries = <s>5</s>

type = <s>crontab</s>
schedule = <s>{"minute": 0}</s>

You’ll notice the configuration options that are dictionaries or have multiple values will be split into their own sections.

To use celerybeat (periodic tasks) you need to declare 1 celerybeat config section per task. The options are:

• type - The type of scheduling your configuration uses, options are crontab, timedelta, and integer.

• schedule - The actual schedule for your type of configuration.

• args - Additional positional arguments.

• kwargs - Additional keyword arguments.

Example configuration for this:

[celerybeat:task1]
type = <s>crontab</s>
schedule = <s>{"minute": 0}</s>

type = <s>timedelta</s>
schedule = <s>{"seconds": 30}</s>
args = <s>[16, 16]</s>

type = <s>crontab</s>
schedule = <s>{"hour": 0, "minute": 0}</s>
kwargs = <s>{"boom": "shaka"}</s>

type = <s>integer</s>
schedule = <s>30</s>

A gotcha you want to watchout for is that the date/time in scheduled tasks is UTC by default. If you want to schedule for an exact date/time for your local timezone you need to set timezone. Documentation for that can be found here:

If you need to find out what timezones are available you can do the following:

from pprint import pprint
from pytz import all_timezones
pprint(all_timezones)

### Worker Execution

The celerybeat worker will read your configuration and schedule tasks in the queue to be executed at the time defined. This means if you are using celerybeat you will end up running 2 workers:

$celery -A pyramid_celery.celery_app worker --ini development.ini$ celery -A pyramid_celery.celery_app beat --ini development.ini

The first command is the standard worker command that will read messages off of the queue and run the task. The second command will read the celerybeat configuration and periodically schedule tasks on the queue.

### Routing

If you would like to route a task to a specific queue you can define a route per task by declaring their queue and/or routing_key in a celeryroute section.

An example configuration for this:

[celeryroute:otherapp.tasks.Task3]
routing_key = <s>turtle</s>

queue = <s>fast_tasks</s>

## Running the worker

To run the worker we just use the standard celery command with an additional argument:

celery worker -A pyramid_celery.celery_app --ini development.ini

If you’ve defined variables in your .ini like %(database_username)s you can use the –ini-var argument, which is a comma separated list of key value pairs:

celery worker -A pyramid_celery.celery_app --ini development.ini --ini-var=database_username=sontek,database_password=OhYeah!

The values in ini-var cannot have spaces in them, this will break celery’s parser.

The reason it is a csv instead of using –ini-var multiple times is because of a bug in celery itself. When they fix the bug we will re-work the API. Ticket is here:

https://github.com/celery/celery/pull/2435

If you use celerybeat scheduler you need to run with the –beat flag to run beat and the worker at the same time.

celery worker --beat -A pyramid_celery.celery_app --ini development.ini

Or you can launch it separately like this:

celery beat -A pyramid_celery.celery_app --ini development.ini

## Logging

If you use the .ini configuration (i.e don’t use celeryconfig.py) then the logging configuration will be loaded from the .ini and will not use the default celery loggers.

You most likely want to add a logging section to your ini for celery as well:

[logger_celery]
level = <s>INFO</s>
handlers =
qualname = <s>celery</s>

and then update your [loggers] section to include it.

If you want use the default celery loggers then you can set CELERYD_HIJACK_ROOT_LOGGER=True in the [celery] section of your .ini.

Celery worker processes do not propagate exceptions inside tasks, but swallow them silently by default. This is related to the behavior of reading asynchronous task results back. To see if your tasks fail you might need to configure celery.worker.job logger to propagate exceptions:

# Make sure Celery worker doesn't silently swallow exceptions
# See http://stackoverflow.com/a/20719461/315168
# https://github.com/celery/celery/issues/2437
[logger_celery_worker_job]
level = <s>ERROR</s>
handlers =
qualname = <s>celery.worker.job</s>
propagate = <s>1</s>

If you want use the default celery loggers then you can set CELERYD_HIJACK_ROOT_LOGGER=True in the [celery] section of your .ini

## Demo

To see it all in action check out examples/long_running_with_tm, run redis-server and then do:

$python setup.py develop$ populate_long_running_with_tm development.ini
$pserve ./development.ini$ celery worker -A pyramid_celery.celery_app --ini development.ini

## 4.0.0

• Drop support for celery 3.0

• Drop support for Pyramid 2.7

• Error out if ini is misconfigured

• Add support for pyramid v2

• Add support for celery v5

• Drop support for old CELERY config options, use only the lower case versions.

• Added support for options in celerybeat.

## 3.0.0

• Support celery 4.0

• Properly handle CELERY_ALWAYS_EAGER

## Project details

Uploaded source
Uploaded py3