All-in-memory job queue with RESTful interface.
Project description
Why restq?
We wanted to have a simple platform independent solution for managing the coordination and distribution of batched execution across our analysis platforms. restq solved our wants into a system that could:
segregate execution based on a category or type (realm),
manage priorities of job execution (ordered queues),
enqueue, check-out, and expiry time based (almost FIFO) dequeuing of jobs from a queue.
status of jobs remaining against arbitrary tag indices.
zero configuration for the concepts talked about above.
What’s in restq:
An implementation of the execution management system described above.
A RESTful web API that exposes complete control over the execution management system.
A Python client that seamlessly interfaces the RESTful web API.
For additional tips / tricks with this restq feel free to post a question at the github restq/issues page.
Project hosting provided by github.com.
[mjdorma+restq@gmail.com] and [sptonkin@outlook.com]
Install and run
Simply run the following:
> python setup.py install
or PyPi:
> pip install restq
Getting started
Coding with restq
A simple example on how
> restq web & > ipython In [1]: from restq import Realms In [2]: realms = Realms() In [3]: realms.test. realms.test.add realms.test.bulk_add realms.test.bulk_flush realms.test.get_job realms.test.get_tag_status realms.test.get_tagged_jobs realms.test.name realms.test.pull realms.test.remove_job realms.test.remove_tagged_jobs realms.test.request realms.test.requester realms.test.set_default_lease_time realms.test.set_queue_lease_time realms.test.status In [3]: realms.test.add('job 1', 0, 'do the dishes', tags=['house work']) In [4]: realms.test.add('job 2', 0, 'cut the grass', tags=['house work']) In [5]: realms.test.add('job 3', 1, 'fix bugs in restq', tags=['devel']) In [6]: realms.test.add('job 4', 3, 'document restq', tags=['devel']) In [7]: realms.test.add('job 5', 0, 'go for walk', tags=['sport']) In [8]: realms.test.status Out[8]: {u'queues': {u'0': 4, u'1': 1, u'2': 1, u'3': 1}, u'total_jobs': 7, u'total_tags': 3} In [9]: jobs = realms.test.pull(count=7) In [10]: jobs Out[10]: {u'job 1': [0, u'do the dishes'], u'job 2': [0, u'cut the grass'], u'job 3': [1, u'fix bugs in restq'], u'job 4': [3, u'document restq'], u'job 5': [0, u'go for walk'], u'job 6': [0, u'go for walk with dog'], u'job 7': [2, u'go for bike ride']} In [11]: realms.test.get_tag_status('house work') Out[11]: {u'count': 2} In [12]: realms.test.get_tagged_jobs('devel') Out[12]: {u'job 3': {u'data': u'fix bugs in restq', u'queues': [[1, 82.17003393173218]], u'tags': [u'devel']}, u'job 4': {u'data': u'document restq', u'queues': [[3, 82.16989994049072]], u'tags': [u'devel']}}
Issues
Source code for restq is hosted on GitHub. Please file bug reports with GitHub’s issues system.
Change log
version 0.0.4 (09/06/2013)
config and cli implementation
version 0.0.3 (06/06/2013)
bulk post & stable error handling
version 0.0.1 (10/04/2013)
pre life
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.