Skip to main content

lightweight Python job queue with multiprocessing support

Project description

lightqueue is a lightweight job queue that processes jobs (Python function calls) from a queue located inside a Redis database.

To install it:

$ pip install lightqueue

After you have lightqueue (and Redis) installed, you are ready to start queueing jobs. Say you have a function like this in a module called mymodule

def myfunc(num):
  print sum([x for x in range(num)])

To add the execution of this function as a job into lightqueue:

>>> from lightqueue.queue import Queue
>>> from mymodule import myfunc
>>> q = Queue()
>>> q.enqueue(myfunc, 9999) # add the job myfunc(9999) to the queue
>>> q.enqueue(myfunc, 1234567) # add the job myfunc(1234567) to the queue

To start processing these jobs, type this in a shell prompt:

$ lightqueue start

Change db server

By default, lightqueue adds jobs to and processes jobs from the Redis server located at localhost:6379 on db=0. To change any of these settings:

>>> Queue q = Queue(host='myredishost', port=7323, db=4)

Then give the lightqueue process the same settings:

$ lightqueue start -host myredishost -port 7323 -db 4

Parallel Processing

By default, lightqueue processes one job from the queue at a time. To process more than one job at once (let’s say 4), start up lightqueue with the following command-line args:

$ lightqueue start -e parallel -workers 4

This uses the Python multiprocessing module so be aware of all of the usual caveats of parallel processing.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
lightqueue-0.1.1.tar.gz (5.9 kB) Copy SHA256 hash SHA256 Source None Apr 4, 2013

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page