collective.taskqueue 0.3.1

UNKNOWN

collective.taskqueue

Yet another way to dispatch and execute asynchronous tasks in Plone.

This is an experiment. Not yet battle tested.

collective.taskqueue enables asynchronous tasks in Plone add-ons by providing a small framework for asynchronously queueing requests for ZPublisher. Even this design may not provide the best performance for your asynchronous Plone tasks, it should be the easiest to use: asynchronous tasks are just normal calls to normally registered browser views (or other traversable callables) and they authenticated using PAS as all the other requests.

Minimal configuration:

zope-conf-additional =
    %import collective.taskqueue
    <taskqueue-server>
    </taskqueue-server>

Minimal configuration gives you one volatile instance-local queue and consumer, but no guarantee on delivery.

Example Redis configuration:

eggs =
    collective.taskqueue [redis]

zope-conf-additional =
    %import collective.taskqueue
    <product-config collective.taskqueue>
      queue redis
      redis_unix_socket_path ${buildout:directory}/var/redis.sock
    </product-config>
    <taskqueue-server>
      name ${:_buildout_section_name_}
      queue redis
    </taskqueue-server>

Redis-support gives you machine-local queues, which can be shared between instances. Only the consuming instance requires <taskqueue-server />. It’s recommended to only use local Redis-installations, because remote connections can be killed by firewalls (there’s no ping or heartbeat to keep the connection alive).

Queue a task:

from collective.taskqueue import taskqueue
taskqueue.add('/Plone/path/to/my/view')

Tasks are queued (and consumed) after a successful transaction.

By default, taskqueue.add copies headers from the current requests to the asynchronous request. That should be enough to authenticate the requests as the same way as the current request was authenticated. More robust authentication can be implemented with a custom PAS-plugin. (A default one may be shipped soon with collective.taskqueue…).

Advanced configuration

Supported <taskqueue-server />-settings are:

name (default=default)

Consumer name, preferably instance name. Consumer name can be used by queues when reserving messages from broker for processing.

queue (default=default)

Queue name for this consumer (consuming server). There must exist a registered utility providing ITaskQueue with this name.

concurrent_limit (default=1)

Maximum concurrent task limit for this consumer. The limit should be less than zserver-thread or just 1.

retry_max_count (default=10)

Maximum ZPublisher retry count for requests dispatched by this consumer. Once the limit has been exceeded, the conflicting task may be permanently skipped, depending the used queue.

Supported Redis-connection options are:

• redis_host

• redis_port

• redis_db

• redis_password

• redis_socket_timeout

• redis_errors

• redis_unix_socket_path (alternative to redis_host with redis_port)

Custom queues can be declared by registering custom utilities:

<utility factory="collective.taskqueue.taskqueue.LocalVolatileTaskQueue"
         name="my-instance-local-volatile-queue" />

<utility factory="collective.taskqueue.redisqueue.RedisTaskQueue"
         name="my-redis-queue" />

Advanced usage

taskqueue.add accepts the following arguments (with default value):

url (required, no default)

Target path representing the task to be called.

method (optional, default=GET)

HTTP-method for the call. Must be either GET or POST.

params (optional, default=None)

A dictionary of optional task arguments, which are appended as query string after the given url. (When params are provided, url must not already include any querystring).

headers (optional, default=None)

A dictionary of optional HTTP-headers to be appended to (or used to replace) the headers copied from the active request.

payload (optional, default=current)

An optional payload for POST-request. Payload from the active request will be copied by default. Copying the active payload can be prevented with payload=None.

queue (optional, default=configured-default)

An optional queue name, when queueing tasks for non-default queues.

Changelog

0.3.1 (2013-11-13)

• Enhance acceptance testing support with the first acceptance tests [Asko Soukka]

0.3.0 (2013-11-10)

• Fix TaskQueueServer to re-connect to Redis after Redis restart [Asko Soukka]

• Fix to ping Redis on Zope start only in development mode [Asko Soukka]

• Add optional Task Queue PAS plugin to independently authenticate queued tasks as their creator [Asko Soukka]

0.2.2 (2013-11-09)

• Fix to flush Redis pub-notifications only when queue has been emptied to ensure that all messages will be processed [Asko Soukka]

0.2.1 (2013-11-09)

• Fix taskqueue socket to be not readable by default [Asko Soukka]

0.2.0 (2013-11-09)

• Enhance Redis-integration to connect redis notification pubsub-socket directly to asyncore on instant message handling [Asko Soukka]

• Fix to require redis >= 2.4.10 [fixes #2] [Asko Soukka]

• Fix to not start with clear error when clearly intending to use RedisTaskQueues without redis-dependencies. Also crash when cannot connect to Redis. [fixes #1] [Asko Soukka]

0.1.0 (2013-11-03)

• First release for experimental use.