lightweight Python job queue with multiprocessing support
Project description
lightqueue is a lightweight job queue that processes jobs (Python function calls) from a queue located inside a Redis database.
To install it:
$ pip install lightqueue
After you have lightqueue (and Redis) installed, you are ready to start queueing jobs. Say you have a function like this in a module called mymodule
def myfunc(num): print sum([x for x in range(num)])
To add the execution of this function as a job into lightqueue:
>>> from lightqueue.queue import Queue >>> from mymodule import myfunc >>> q = Queue() >>> q.enqueue(myfunc, 9999) # add the job myfunc(9999) to the queue >>> q.enqueue(myfunc, 1234567) # add the job myfunc(1234567) to the queue
To start processing these jobs, type this in a shell prompt:
$ lightqueue start
Change db server
By default, lightqueue adds jobs to and processes jobs from the Redis server located at localhost:6379 on db=0. To change any of these settings:
>>> Queue q = Queue(host='myredishost', port=7323, db=4)
Then give the lightqueue process the same settings:
$ lightqueue start -host myredishost -port 7323 -db 4
Parallel Processing
By default, lightqueue processes one job from the queue at a time. To process more than one job at once (let’s say 4), start up lightqueue with the following command-line args:
$ lightqueue start -e parallel -workers 4
This uses the Python multiprocessing module so be aware of all of the usual caveats of parallel processing.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.