This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Project Description

Introduction

This recipe installs Celery and creates a celeryconfig.py module with the specified configuration options. It helps managing multiple configurations (e.g. development and production) using buildout.

You can use it in a part like this:

[celery]
recipe = collective.recipe.celery
broker-transport = sqlakombu.transport.Transport
broker-host = sqlite:///celery_broker.db
result-backend = database
result-dburi = sqlite:///celery_results.db
imports = myapp.tasks
eggs =
    kombu-sqlalchemy
    myapp

Supported options

General options

eggs
A list of additional eggs you want to make available to Celery. Use this to add additional dependencies such as kombu-sqlalchemy or the module(s) containing your task definitions.
scripts
Controls which scripts are generated. If the option is omitted, then all scripts will be generated. If no value is given, then script generation is disabled.
config-path

The location of the directory containing the celeryconfig.py module. By default the config module is created in the part directory. You can use this in other parts to include the config module:

[celery]
recipe = collective.recipe.celery

[myapp]
recipe = zc.recipe.egg
eggs = myapp
extra-paths = ${celery:config-path}

Celery options

The following configuration options are supported. See Celery documentation for more details.

broker-transport
The Kombu transport to use. You can use a custom transport class name, or select one of the built-in transports: amqplib, pika, redis, beanstalk, sqlalchemy, django, mongodb, couchdb.
broker-host
The hostname of the broker.
broker-port
The port number of the broker.
broker-user
The username to connect as.
broker-password
The password to connect with.
broker-vhost
The virtual host.
result-backend
The backend used to store task results. Can be one of database, cache, mongodb, redis, tyrant or amqp.
result-dburi
Connection string for the database result backend.
imports
A list of modules to import when the celery daemon starts. Specify one module per line.
celeryd-log-file
The filename where the celery daemon logs messages to.
celeryd-log-level
The log level, can be one of DEBUG, INFO, WARNING, ERROR or CRITICAL.
celeryd-concurrency
The number of concurrent worker processes/threads/green threads, executing tasks.
additional-config

Any additional configuration directives can be added using the additional-config option.

Example:

additional-config =
    CELERY_TASK_PUBLISH_RETRY=True
    CELERY_TASK_PUBLISH_RETRY_POLICY={"max_retries": 2,
                                      "interval_start": 10,
                                      "interval_step": 0,
                                      "interval_max": 10}

Changelog

1.0 (2011-08-15)

  • Initial release. [buchi]

Example usage

We’ll start by creating a buildout that uses the recipe:

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [celery]
... recipe = collective.recipe.celery
... broker-transport = sqlakombu.transport.Transport
... broker-host = sqlite:///celery_broker.db
... result-backend = database
... result-dburi = sqlite:///celery_results.db
... imports = myapp.tasks
... """% dict(server=link_server))

Running the buildout gives us:

>>> print system(buildout)
Installing celery.
celery: Creating directory /sample-buildout/parts/celery.
celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py.
Getting distribution for 'celery'.
Got celery 2.3.1.
Generated script '/sample-buildout/bin/celeryctl'.
Generated script '/sample-buildout/bin/celeryd'.
<BLANKLINE>

Check that we have the celery scripts:

>>> ls(sample_buildout, 'bin')
-  buildout
-  celeryctl
-  celeryd

Check that we got a celery config file:

>>> ls(sample_buildout, 'parts', 'celery')
- celeryconfig.py

If we run the celeryd script, it prints out the config data:

>>> print(system(join(sample_buildout, 'bin', 'celeryd')))
BROKER_HOST='sqlite:///celery_broker.db'
BROKER_TRANSPORT='sqlakombu.transport.Transport'
CELERY_IMPORTS=('myapp.tasks',)
CELERY_RESULT_BACKEND='database'
CELERY_RESULT_DBURI='sqlite:///celery_results.db'
<BLANKLINE>

We can include additional eggs using the eggs option:

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [celery]
... recipe = collective.recipe.celery
... eggs =
...     other
... """% dict(server=link_server))

>>> print system(buildout),
Uninstalling celery.
Installing celery.
celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py.
Getting distribution for 'other'.
Got other 1.0.
Generated script '/sample-buildout/bin/celeryctl'.
Generated script '/sample-buildout/bin/celeryd'.

We can control which scripts are generated using the scripts option. If no value is given, then script generation is disabled:

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [celery]
... recipe = collective.recipe.celery
... scripts =
... """% dict(server=link_server))

>>> print system(buildout),
Uninstalling celery.
Installing celery.
celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py.

>>> ls(sample_buildout, 'bin')
-  buildout

Let’s create the celeryd script only:

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [celery]
... recipe = collective.recipe.celery
... scripts =
...     celeryd
... """% dict(server=link_server))

>>> print system(buildout),
Uninstalling celery.
Installing celery.
celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py.
Generated script '/sample-buildout/bin/celeryd'.

>>> ls(sample_buildout, 'bin')
-  buildout
-  celeryd

The supported configuration directives may be of various types including strings, integers and tuples:

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [celery]
... recipe = collective.recipe.celery
... broker-port = 8080
... broker-user = guest
... imports =
...     myapp.tasks
...     other.tasks
... """% dict(server=link_server))

>>> print system(buildout),
Uninstalling celery.
Installing celery.
celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py.
Generated script '/sample-buildout/bin/celeryctl'.
Generated script '/sample-buildout/bin/celeryd'.

Let’s verify the generated config data:

>>> cat(sample_buildout, 'parts', 'celery', 'celeryconfig.py')
BROKER_PORT = 8080
BROKER_USER = 'guest'
CELERY_IMPORTS = ('myapp.tasks', 'other.tasks')
<BLANKLINE>

The recipe supports a limited set of celery’s configuration directives. Any additional directives can be added using the additional-config option:

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [celery]
... recipe = collective.recipe.celery
... additional-config =
...     CELERY_TASK_PUBLISH_RETRY = True
...     CELERY_TASK_PUBLISH_RETRY_POLICY = {"max_retries": 2,
...                                         "interval_start": 10,
...                                         "interval_step": 0,
...                                         "interval_max": 10}
... """% dict(server=link_server))

>>> print system(buildout),
Uninstalling celery.
Installing celery.
celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py.
Generated script '/sample-buildout/bin/celeryctl'.
Generated script '/sample-buildout/bin/celeryd'.

Let’s verify the generated config data:

>>> cat(sample_buildout, 'parts', 'celery', 'celeryconfig.py')
CELERY_TASK_PUBLISH_RETRY = True
CELERY_TASK_PUBLISH_RETRY_POLICY = {"max_retries": 2,
"interval_start": 10,
"interval_step": 0,
"interval_max": 10}
<BLANKLINE>
Release History

Release History

1.0

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
collective.recipe.celery-1.0.tar.gz (17.4 kB) Copy SHA256 Checksum SHA256 Source Aug 15, 2011

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting