lightweight Python job queue with multiprocessing support
lightqueue is a lightweight job queue that processes jobs (Python function calls) from a queue located inside a Redis database.
To install it:
$ pip install lightqueue
After you have lightqueue (and Redis) installed, you are ready to start queueing jobs. Say you have a function like this in a module called mymodule
def myfunc(num): print sum([x for x in range(num)])
To add the execution of this function as a job into lightqueue:
>>> from lightqueue.queue import Queue >>> from mymodule import myfunc >>> q = Queue() >>> q.enqueue(myfunc, 9999) # add the job myfunc(9999) to the queue >>> q.enqueue(myfunc, 1234567) # add the job myfunc(1234567) to the queue
To start processing these jobs, type this in a shell prompt:
$ lightqueue start
Change db server
By default, lightqueue adds jobs to and processes jobs from the Redis server located at localhost:6379 on db=0. To change any of these settings:
>>> Queue q = Queue(host='myredishost', port=7323, db=4)
Then give the lightqueue process the same settings:
$ lightqueue start -host myredishost -port 7323 -db 4
By default, lightqueue processes one job from the queue at a time. To process more than one job at once (let’s say 4), start up lightqueue with the following command-line args:
$ lightqueue start -e parallel -workers 4
This uses the Python multiprocessing module so be aware of all of the usual caveats of parallel processing.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.