Skip to main content

lightweight Python job queue with multiprocessing support

Project description

lightqueue is a lightweight job queue that processes jobs (Python function calls) from a queue located inside a Redis database.

To install it:

$ pip install lightqueue

After you have lightqueue (and Redis) installed, you are ready to start queueing jobs. Say you have a function like this in a module called mymodule

def myfunc(num):
  print sum([x for x in range(num)])

To add the execution of this function as a job into lightqueue:

>>> from lightqueue.queue import Queue
>>> from mymodule import myfunc
>>> q = Queue()
>>> q.enqueue(myfunc, 9999) # add the job myfunc(9999) to the queue
>>> q.enqueue(myfunc, 1234567) # add the job myfunc(1234567) to the queue

To start processing these jobs, type this in a shell prompt:

$ lightqueue start

Change db server

By default, lightqueue adds jobs to and processes jobs from the Redis server located at localhost:6379 on db=0. To change any of these settings:

>>> Queue q = Queue(host='myredishost', port=7323, db=4)

Then give the lightqueue process the same settings:

$ lightqueue start -host myredishost -port 7323 -db 4

Parallel Processing

By default, lightqueue processes one job from the queue at a time. To process more than one job at once (let’s say 4), start up lightqueue with the following command-line args:

$ lightqueue start -e parallel -workers 4

This uses the Python multiprocessing module so be aware of all of the usual caveats of parallel processing.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lightqueue-0.1.1.tar.gz (5.9 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page