lightqueue 0.1.1

lightweight Python job queue with multiprocessing support

lightqueue is a lightweight job queue that processes jobs (Python function calls) from a queue located inside a Redis database.

To install it:

$ pip install lightqueue

After you have lightqueue (and Redis) installed, you are ready to start queueing jobs. Say you have a function like this in a module called mymodule

def myfunc(num):
  print sum([x for x in range(num)])

To add the execution of this function as a job into lightqueue:

>>> from lightqueue.queue import Queue
>>> from mymodule import myfunc
>>> q = Queue()
>>> q.enqueue(myfunc, 9999) # add the job myfunc(9999) to the queue
>>> q.enqueue(myfunc, 1234567) # add the job myfunc(1234567) to the queue

To start processing these jobs, type this in a shell prompt:

$ lightqueue start

Change db server

By default, lightqueue adds jobs to and processes jobs from the Redis server located at localhost:6379 on db=0. To change any of these settings:

>>> Queue q = Queue(host='myredishost', port=7323, db=4)

Then give the lightqueue process the same settings:

$ lightqueue start -host myredishost -port 7323 -db 4

Parallel Processing

By default, lightqueue processes one job from the queue at a time. To process more than one job at once (let’s say 4), start up lightqueue with the following command-line args:

$ lightqueue start -e parallel -workers 4

This uses the Python multiprocessing module so be aware of all of the usual caveats of parallel processing.