md.recipe.celery 1.1

Buildout recipe for installing celery for use with Zope's ZCA and configuring it using an ini file.

Introduction
============

Buildout recipe for installing Celery for use with Zope's ZCA and configuring
it using an ini file.

This recipe was originally based on collective.recipe.celery,
but differs greatly so that it now has become its own package. This recipe
defines a custom loader that can read celery configuration from an ini file.
This allows you to generate the configuration by using a template for
example, using collective.recipe.template. The custom loader also bootstraps
the ZCA automatically when a worker initializes.

You can use it in a part like this::

[celery]
recipe = md.recipe.celery
eggs =
${buildout:eggs}
celery[redis]
celery_conf = etc/celery.ini
logging_conf = etc/logging.ini
zope_conf = etc/zope.conf


Supported options
=================

General options
---------------

eggs
A list of additional eggs you want to make available to Celery. Use this to
add additional dependencies such as ``kombu-sqlalchemy`` or the module(s)
containing your task definitions.

celery_conf
Location of the Celery configuration .ini file.

logging_conf
Location of the file containing the python logging setup

zope_conf
Location of the zope configuration file.


Configuration
=============

Example of an ini file that could be used by this recipe::

[celery]
BROKER_URL = sqlite:///celery_broker.db
CELERY_IMPORTS = myapp.tasks,otherapp.tasks
CELERY_RESULT_BACKEND = sqlite:///celery_results.db
CELERY_RESULT_SERIALIZER = json

[celerybeat:task1]
# Execute every hour
task = myapp.tasks.Task1
type = crontab
schedule = {"minute": 0}

[celerybeat:task2]
# Execute every 30 seconds
task = myapp.tasks.Task2
type = timedelta
schedule = {"seconds": 30}

[celerybeat:task3]
# Execute at midnight
task = otherapp.tasks.Task3
type = crontab
schedule = {"hour": 0, "minute": 0}

The [celery] section defines all the celery options. Every [celerybeat]
section defines an individual task.

Example usage
=============

Create a celery ini file holding your celery configuration::

>>> write(sample_buildout, 'celery.ini',
... """
... [celery]
... BROKER_URL = sqlite:///celery_broker.db
... CELERY_IMPORTS = myapp.tasks,otherapp.tasks
... CELERY_RESULT_BACKEND = sqlite:///celery_results.db
... CELERY_RESULT_SERIALIZER = json
...
... [celerybeat:task1]
... # Execute every hour
... task = myapp.tasks.Task1
... type = crontab
... schedule = {"minute": 0}
...
... [celerybeat:task2]
... # Execute every 30 seconds
... task = myapp.tasks.Task2
... type = timedelta
... schedule = {"seconds": 30}
...
... [celerybeat:task3]
... # Execute at midnight
... task = otherapp.tasks.Task3
... type = crontab
... schedule = {"hour": 0, "minute": 0}""")

This file could be generated by a template such as collective.recipe.template.

Next create a logging config::

>>> write(sample_buildout, 'logging.ini',
... """
... [loggers]
... keys = root
...
... [handlers]
... keys = console
...
... [formatters]
... keys = generic
...
... [logger_root]
... level = INFO
... handlers = console
...
... [handler_console]
... class = StreamHandler
... args = (sys.stderr,)
... level = NOTSET
... formatter = generic
...
... [formatter_generic]
... format = %(asctime)s %(levelname)s [%(name)s] %(message)s""")

Then we'll create a buildout that uses the recipe::

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [celery]
... recipe = md.recipe.celery
... celery_conf = celery.ini
... logging_conf = logging.ini
... zope_conf = zope.conf
... """% dict(
... server=link_server))

Running the buildout gives us::

>>> print system(buildout)
Installing celery.
celery: Creating directory /sample-buildout/parts/celery.
celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.
Getting distribution for 'celery'.
Got celery 2.3.1.
Generated script '/sample-buildout/bin/celeryctl'.
Generated script '/sample-buildout/bin/celeryd'.
<BLANKLINE>

Check that we have the celery scripts::

>>> ls(sample_buildout, 'bin')
- buildout
- celeryctl
- celeryd

Check that we got the celery loaders::

>>> ls(sample_buildout, 'parts', 'celery')
- loader.py
- zopeloader.py

If we run the celeryd script, it prints out the config data::

>>> print(system(join(sample_buildout, 'bin', 'celeryctl')))
[('BROKER_URL', 'sqlite:///celery_broker.db'),
('CELERYBEAT_SCHEDULE',
{'task1': {'schedule': <crontab: 0 * * * * (m/h/d/dM/MY)>,
'task': 'myapp.tasks.Task1'},
'task2': {'schedule': datetime.timedelta(0, 30),
'task': 'myapp.tasks.Task2'},
'task3': {'schedule': <crontab: 0 0 * * * (m/h/d/dM/MY)>,
'task': 'otherapp.tasks.Task3'}}),
('CELERYD_HIJACK_ROOT_LOGGER', False),
('CELERY_IMPORTS', ['myapp.tasks', 'otherapp.tasks']),
('CELERY_RESULT_BACKEND', 'sqlite:///celery_results.db'),
('CELERY_RESULT_SERIALIZER', 'json')]
<BLANKLINE>

We can include additional eggs using the eggs option::

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [celery]
... recipe = md.recipe.celery
... eggs =
... other
... """% dict(server=link_server))

>>> print system(buildout),
Uninstalling celery.
Installing celery.
celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.
Getting distribution for 'other'.
Got other 1.0.
Generated script '/sample-buildout/bin/celeryctl'.
Generated script '/sample-buildout/bin/celeryd'.
Generated script '/sample-buildout/bin/distutilsscript'.

The recipe should handle updates as well, trigger this by pinning celery to
another version and checking one of the outputted scripts.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [versions]
... celery = 2.3.0
...
... [celery]
... recipe = md.recipe.celery
... eggs =
... other
... """% dict(server=link_server))

>>> print system(buildout),
Updating celery.
celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.
Getting distribution for 'celery==2.3.0'.
Got celery 2.3.0.
Generated script '/sample-buildout/bin/celeryctl'.
Generated script '/sample-buildout/bin/celeryd'.
Generated script '/sample-buildout/bin/distutilsscript'.

>>> cat(sample_buildout, 'bin', 'celeryctl')
#!/Users/rwendt/Projects/python/bin/python
<BLANKLINE>
import sys
sys.path[0:0] = [
'/sample-buildout/eggs/celery-2.3.0-py2.7.egg',
'/sample-buildout/eggs/other-1.0-py2.7.egg',
'/sample-buildout/parts/celery',
]
***

We can control which scripts are generated using the scripts option.
If no value is given, then script generation is disabled::

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [celery]
... recipe = md.recipe.celery
... scripts =
... """% dict(server=link_server))

>>> print system(buildout),
Uninstalling celery.
Installing celery.
celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.

>>> ls(sample_buildout, 'bin')
- buildout

Let's create the celeryd script only::

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = celery
... index = %(server)s/index
... find-links = %(server)s
...
... [celery]
... recipe = md.recipe.celery
... scripts =
... celeryd
... """% dict(server=link_server))

>>> print system(buildout),
Uninstalling celery.
Installing celery.
celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.
Generated script '/sample-buildout/bin/celeryd'.

>>> ls(sample_buildout, 'bin')
- buildout
- celeryd

=======
CHANGES
=======

1.1 (2013-12-19)
================

- Add missing files to Manifest.


1.0 (2013-12-18)
================

- Initial release. Based loosely on collective.recipe.celery.