torque 0.4.5

A web hook task queue based on tornado and redis

Overview

Torque is a web hook task queue based on Tornado and redis. It’s designed to solve two problems in the context of a web application:

1. you want to do something later

2. you want to do a number of things in parallel

There are many ways of approaching these problems. For example, in python, you might look at Twisted, Celery and Stackless.

Torque is inspired by Google App Engine’s taskqueue, which models tasks as webhooks. This approach allows you to handle tasks within your normal web application environment by writing request handlers, just as you would to handle a user initiated request.

To use it, you need to run:

1. a redis database

2. ./bin/torque-serve, which exposes a Tornado application (by default on http://localhost:8889)

3. one ./bin/torque-process per queue

Or, if you’re just using the default queue, you can replace ./bin/torque-serve and ./bin/torque-process with:

1. ./bin/torque-run

Which runs the web application and the process loop in seperate threads.

You can process queues ad infinitum, or until they are empty. See torque.process.QueueProcessor.__doc__ for the details.

You can add tasks to the queue in two ways:

1. by posting an HTTP request to the Tornado application run by ./bin/torque-serve

2. or by using the python client api in torque.client

This first method allows you to use Torque from any programming language. The second makes it much simpler if you’re using python.

To add a task using an HTTP request, post to /add_task with two params:

• url which is the url to the webhook you want the task to request

• params which is a json encoded dictionary of the params you want to post to the webhook you’re requesting

An example in python (with the Tornado application available on localhost, running on port 8889) would be:

import json
import urllib

mytask = {
    'url': 'http://mywebservice.com/hooks/do/foo',
    'params': json.dumps({'foo', 'somevalue', 'baz': 99})
}
target_url = 'http://localhost:8889/add_task'
urllib.urlopen(target_url, urllib.urlencode(mytask))

This queued a POST request to http://mywebservice.com/hooks/do/foo with the params foo=somevalue and baz=99 to be made as soon as possible.

You can do something similar using any programming language that can make url requests. However, if you are using python, it’s much simpler to use the client api that Torque provides:

from torque.client import add_task
t = add_task(url='http://mywebservice.com/hooks/do/foo', params={'a': 1})

Note that this doesn’t require json encoding the params. For all the client api options, see torque.client.Task.__doc__.

Individual tasks backoff exponentially if they error, upto a maximum backoff delay that’s configurable as --max_task_delay, until they error --max_task_errors times (at which point they get deleted).

Install

Install the redis and Tornado dependencies. Then install Torque:

$ easy_install torque

Or manually from source:

$ git clone git://github.com/thruflo/torque.git
$ cd torque
$ python setup.py install

Run

Run redis:

$ ./redis-server

Start the Tornado application:

$ ./bin/torque-serve

If you want to run the tests, use:

$ ./bin/nosetests -w ./src/torque --with-doctest
.......
----------------------------------------------------------------------
Ran 7 tests in 22.627s

OK

Start the default task queue running ad infinitum:

$ ./bin/torque-process

See --help against either of the torque console scripts for a list of configuration options. For example, to run a second queue called foobar, you might use:

./bin/torque-process --queue_name=foobar

Or to process the default queue once until empty you might use:

./bin/torque-process --finish_on_empty=true --max_task_errors=3

Or to do exactly the same from python code:

from torque.processor import QueueProcessor
QueueProcessor(max_task_errors=3).process(finish_on_empty=true)

Read the source code for more information.