shredder 0.5

Simple multiprocessing

An easy way to distribute work to processes and aggregate results.

Installation

To install shredder:

$ sudo pip install shredder

or from source:

$ sudo python setup.py install

Getting Started

>>> from shredder import Shredder
>>>
>>>
>>> function work_generator:
>>>     with open(f) as f:
>>>         yield readline(f)
>>>
>>> function worker(shredder, data):
>>>    results = do_something_with_data(data)
>>>    return results
>>>
>>> function aggregator(results):
>>>    dosomethingwitresults
>>>
>>>  shredder = Shredder(
>>>         work_generator=work_generator,
>>>         worker=worker,
>>>         aggregator=aggregator,
>>>         num_processes=5,
>>>  )
>>>
>>>  shredder.start()
>>>