Skip to main content

AMQP based worker/master pattern framework

Project description

AMQP based RPC library for Tornado

Use cases

  • Task queue
  • Load balancing for different CPU-bound HTTP handlers
  • … other systems which involve RPC

Installation

pip install crew

Example

See the full example.

Usage

For example create your first app, and save as master.py:

import tornado.ioloop
import tornado.gen
import tornado.web
import tornado.log
import json
import tornado.options
from crew import TimeoutError, ExpirationError
from crew.master.tornado import Client

class MainHandler(tornado.web.RequestHandler):

    @tornado.gen.coroutine
    def get(self):
        resp = yield self.settings['crew'].call(None, priority=100)
        self.write(json.dumps(resp))

cl = Client()
application = tornado.web.Application(
    [
        (r"/", MainHandler),
    ],
    crew=cl,
    autoreload=True,
    debug=True,
)

if __name__ == "__main__":
    cl.connect()
    tornado.options.parse_command_line()
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

And create your first task, and save as worker.py:

# encoding: utf-8
from crew.worker import run, context, Task

@Task('test')
def long_task(req):
    context.settings.counter += 1
    return {"counter": context.settings.counter}

run(
    counter=0
)

After that run it:

$ python master.py &
$ python worker.py &
$ wait

Try to test it:

$ curl http://localhost:8888/

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for crew, version 0.9.2
Filename, size File type Python version Upload date Hashes
Filename, size crew-0.9.2.tar.gz (11.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page