Skip to main content
Help us improve Python packaging – donate today!

Mongo Redis Queue

Project Description

MRQ

Mongo Redis Queue - A distributed worker task queue in Python

/!MRQ is not yet ready for public use. Soon!

Why?

MRQ is an opinionated task queue. It aims to be simple and beautiful like http://python-rq.org while having performance close to http://celeryproject.org

MRQ was first developed at http://pricingassistant.com and its initial feature set matches the needs of worker queues with heterogenous jobs (IO-bound & CPU-bound, lots of small tasks & a few large ones).

The main features of MRQ are:

  • Simple code: We originally switched from Celery to RQ because Celery’s code was incredibly complex and obscure ([Slides](http://www.slideshare.net/sylvinus/why-and-how-pricing-assistant-migrated-from-celery-to-rq-parispy-2)). MRQ should be as easy to understand as RQ and even easier to extend.
  • Great dashboard: Have visibility and control on everything: queued jobs, current jobs, worker status, …
  • Per-job logs: Get the log output of each task separately in the dashboard
  • Gevent worker: IO-bound tasks can be done in parallel in the same UNIX process for maximum throughput
  • Supervisord integration: CPU-bound tasks can be split across several UNIX processes with a single command-line flag
  • Job management: You can retry, requeue, cancel jobs from the code or the dashboard.
  • Performance: Bulk job queueing, easy job profiling
  • Easy configuration: Every aspect of MRQ is configurable through command-line flags or a configuration file
  • Job routing: Like Celery, jobs can have default queues, timeout and ttl values.
  • Thorough testing: Edge-cases like worker interrupts, Redis failures, … are tested inside a Docker container.
  • Builtin scheduler: Schedule tasks by interval or by time of the day
  • Greenlet tracing: See how much time was spent in each greenlet to debug CPU-intensive jobs.

Performance

On a MacbookPro, we see 1300 jobs/second in a single worker process with very simple jobs that store results, to measure the overhead of MRQ. However what we are really measuring there is MongoDB’s write performance.

Tests

Testing is done inside a Docker container for maximum repeatability. We don’t use Travis-CI or friends because we need to be able to kill our process dependencies (MongoDB, Redis, …) on demand.

Therefore you need to ([install docker](https://www.docker.io/gettingstarted/#h_installation)) to run the tests. If you’re not on an os that supports natively docker, don’t forget to start up your VM and ssh into it.

` $ make test `

You can also open a shell inside the docker (just like you would enter in a virtualenv) with:

` $ make docker (if it wasn't build before) $ make ssh `

Use in your application

add MRQ to your environment

PyPy

Earlier in its development MRQ was tested successfully on PyPy but we are waiting for better PyPy+gevent support to continue working on it, as performance was worse than CPython.

TODO

alpha

  • Max Retries
  • MongoDB/Redis disconnect tests in more contexts (long-running queries, …)

public beta

  • Full linting
  • Code coverage
  • Public docs
  • PyPI

whishlist

  • task progress
  • ETAs / Lag stats for each queue + graphes
  • uniquestarted/uniquequeued via bulk sets?
  • Base cleaning/retry tasks: move
  • Current greenlet traces in dashboard
  • Move monitoring in a thread to protect against CPU-intensive tasks
  • Bulk queues
  • Tasksets
  • Full PyPy support
  • Search in dashboard

Credits

Inspirations:
  • RQ
  • Celery
JS libraries used in the Dashboard:

… as well as all the Python modules in requirements.txt!

Release history Release notifications

History Node

0.2.1

History Node

0.2.0

History Node

0.1.18

History Node

0.1.17

History Node

0.1.16

History Node

0.1.15

History Node

0.1.14

History Node

0.1.13

History Node

0.1.12

History Node

0.1.11

History Node

0.1.10

History Node

0.1.9

History Node

0.1.8

History Node

0.1.7

History Node

0.1.6

History Node

0.1.5

History Node

0.1.4

History Node

0.1.3

History Node

0.1.2

History Node

0.1.1

History Node

0.1.0

History Node

0.0.48

History Node

0.0.47

History Node

0.0.46

History Node

0.0.45

History Node

0.0.44

History Node

0.0.43

History Node

0.0.42

History Node

0.0.41

History Node

0.0.40

History Node

0.0.39

History Node

0.0.38

History Node

0.0.37

History Node

0.0.36

History Node

0.0.35

History Node

0.0.34

History Node

0.0.33

History Node

0.0.32

History Node

0.0.31

History Node

0.0.30

History Node

0.0.29

History Node

0.0.28

History Node

0.0.27

History Node

0.0.26

History Node

0.0.25

History Node

0.0.24

History Node

0.0.23

History Node

0.0.22

History Node

0.0.21

History Node

0.0.20

History Node

0.0.19

History Node

0.0.18

History Node

0.0.17

History Node

0.0.16

History Node

0.0.15

History Node

0.0.14

History Node

0.0.13

History Node

0.0.12

History Node

0.0.11

History Node

0.0.10

History Node

0.0.9

History Node

0.0.8

History Node

0.0.7

This version
History Node

0.0.6

History Node

0.0.5

History Node

0.0.4

History Node

0.0.3

History Node

0.0.2

History Node

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
mrq-0.0.6.tar.gz (21.9 kB) Copy SHA256 hash SHA256 Source None Mar 4, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page