rotterdam 0.0.6

UNKNOWN

rotterdam
=========

Rotterdam is a simple distributed job queue system, designed to be as simple as possible
and with as few dependencies as possible.

It uses [redis](http://redis.io/) as its datastore and is heavily inspired by the [unicorn](http://unicorn.bogomips.org)
and [gunicorn](https://github.com/benoitc/gunicorn) master/worker process model.

## Design



## Installation

Rotterdam is available via pypi, installing is as easy as:
```
pip install rotterdam
```

## Usage

Two executables are provided, `rotterdam` and `rotterdamctl`.


### Starting up
To start rotterdam, run the `rotterdam` executable and pass in the location
of a config file (an example.cfg) is included in this here repo:

```
[ ~ ] $ rotterdam example.cfg
INFO:rotterdam.master:Starting master (52174)
INFO:rotterdam.master:Listening on port 8765
INFO:rotterdam.master:Starting up worker
INFO:rotterdam.master:Starting up worker
```

### Sending jobs
All a client program has to do is instantiate a `Client` class with the proper host
and port and call `enqueue`:
```python
from rotterdam import Client

client = Client("localhost") # default port is 8765

client.enqueue("rotterdam.example:some_job", "thingy", "guy", foo="bar")
client.enqueue("rotterdam.example:some_job", "derp", "hork", foo="bazz")
```
The first argument to `enqueue` is the full namespace of the job to run
and the rest are passed on to the job itself.

In this example, the job is a simple function that prints out its own arguments:
```python
import time

def some_job(arg1, arg2, foo=None):
time.sleep(2)
print "arg1: %s" % arg1
print "arg2: %s" % arg2
print "foo: %s" % foo
```
So once the client program is run the rotterdam process will print out the args
on its end:
```
arg1: derp
arg2: hork
foo: bazz
arg1: thingy
arg2: guy
foo: bar
```
Note that since it's jobs are executed _concurrently_ with worker processes they
don't necessarily execute in the same order the client sends them.

## Controlling the master process
Rotterdam uses inter-process communcation (IPC) signals for most tasks so that
the master/worker processes can chug along the whole time without needed to
be restarted. The `rotterdamctl` program is a handy utility for sending
the proper signals to the proper process. This program also takes the location
of a config file as the first argument. Make sure to use the same config file
as the rotterdam process you want to control!

### Controlling the number of workers
To add a worker to the existing rotterdam processes, pass the `expand` command
to `rotterdamctl`:
```
[ ~ ] $ rotterdamctl example.cfg expand
```
The master processes will log that a new worker is added:
```
INFO:rotterdam.master:Upping number of workers to 3
INFO:rotterdam.master:Starting up worker
```
Contracting the number of workers is a similiar process, but with the `contract`
command:
```
[ ~ ] $ rotterdamctl example.cfg contract
```
```
INFO:rotterdam.master:Contracting number of workers to 2
INFO:rotterdam.master:Worker exiting
```
### Reloading configuration settings
The rotterdam master process has a facility for reloading its config file on-the-fly
so no work is lost. It is invoked with the `reload` command to `rotterdamctl`.
```
[ ~ ] $ rotterdamctl example.cfg reload
```
The master process will then re-read the config file and signal each worker process
to wrap up whatever it's doing while at the same time spawning new worker processes
based on the new config.
```
INFO:rotterdam.master:Reloading config
INFO:rotterdam.master:Starting up worker
INFO:rotterdam.master:Starting up worker
INFO:rotterdam.master:Worker exiting
INFO:rotterdam.master:Worker exiting
```
### Reloading new code
Naturally, rotterdam only knows of the jobs available to its python runtime. What to
do when you update the code to have shiny new jobs, but you don't want to shut down
or pause any work while updating? For this case there's the `relaunch` command:
```
[ ~ ] $ rotteramctl example.cfg relaunch
```
This little trick comes from gunicorn itself. The master process keeps track of how
it was invoked and when it receives this signal will spawn a brand new process to
replace itself.
```
INFO:rotterdam.master:Winding down old master
INFO:rotterdam.master:Starting master (52580)
INFO:rotterdam.master:Listening on port 8765
INFO:rotterdam.master:Starting up worker
INFO:rotterdam.master:Starting up worker
INFO:rotterdam.master:Worker exiting
INFO:rotterdam.master:Worker exiting
[ ~ ] $
```
Once the new master is up and running, the old master process signals its child worker
processes to wrap up what they're doing and shuts itself down while the new master
processes chugs along and accepts data on the same socket but with freshly-loaded
python code. Clever, eh?