Skip to main content

A Buildout recipe to install and configure Celery for Birdhouse.

Project description

Travis Build

Introduction

birdhousebuilder.recipe.celery is a Buildout recipe to install and configure Celery Distributed Task Queue with Anaconda. This recipe is used by the Birdhouse project.

The recipe is based on collective.recipe.celery.

Usage

The recipe requires that Anaconda is already installed. It assumes that the default Anaconda location is in your home directory ~/anaconda. Otherwise you need to set the ANACONDA_HOME environment variable or the Buildout option anaconda-home.

It installs the celery package from a conda channel in a conda enviroment named birdhouse. The location of the birdhouse environment is .conda/envs/birdhouse. It deploys a Supervisor configuration for Celery in ~/.conda/envs/birdhouse/etc/supervisor/conf.d/celery.conf. Supervisor can be started with ~/.conda/envs/birdhouse/etc/init.d/supervisord start.

The recipe depends on birdhousebuilder.recipe.conda and birdhousebuilder.recipe.supervisor.

Supported options

This recipe supports the following options:

anaconda-home

Buildout option with the root folder of the Anaconda installation. Default: $HOME/anaconda. The default location can also be set with the environment variable ANACONDA_HOME. Example:

export ANACONDA_HOME=/opt/anaconda

Search priority is:

  1. anaconda-home in buildout.cfg
  2. $ANACONDA_HOME
  3. $HOME/anaconda
app
The application instance to use for the celery worker.
eggs
A list of additional eggs you want to make available to Celery. Use this to add additional dependencies and the module(s) containing your task definitions.
use-celeryconfig
Generate and use the celeryconfig.py. Set to false if Celery is configured e.a. by pyramid_celery. Default: true.
use-monitor
If true then Flower is started to monitor Celery. Default: false.

Celery configuration options

The following configuration options are supported. See Celery documentation for more details.

broker-url
The url of the broker. Default: redis://localhost:6379/0
celery-result-backend
The url of the backend used to store task results. Default: redis://localhost:6379/0
celery-imports
List of modules to import when celery starts.
celeryd-concurrency
The number of concurrent worker processes/threads/green threads executing tasks. Defaults to the number of available CPUs.
loglevel
Logging level, choose between DEBUG, INFO, WARNING, ERROR, CRITICAL, or FATAL. Default: WARNING

Example usage

The following example buildout.cfg installs Celery with Anaconda and default options:

[buildout]
parts = celery

anaconda-home = /home/myself/anaconda

[celery]
recipe = birdhousebuilder.recipe.celery
app = mystuff_app
eggs = mysuff

The next example shows a configuration with pyramid_celery and enabled Flower monitoring:

[buildout]
parts = celery

anaconda-home = /home/myself/anaconda

[celery]
recipe = birdhousebuilder.recipe.celery
app = pyramid_celery.celery_app --ini default.ini
eggs = mysuff
use-celeryconfig = false
use-monitor = true

Authors

Carsten Ehbrecht ehbrecht at dkrz.de

Changes

0.1.1 (2015-12-10)

  • Renamed celeryconfig.py template.

0.1.0 (2015-12-10)

  • Initial Release.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for birdhousebuilder.recipe.celery, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size birdhousebuilder.recipe.celery-0.1.1-py2-none-any.whl (11.8 kB) File type Wheel Python version py2 Upload date Hashes View hashes
Filename, size birdhousebuilder.recipe.celery-0.1.1.tar.gz (7.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page