Simple multiprocessing
Project description
An easy way to distribute work to processes and aggregate results.
Installation
To install shredder:
$ sudo pip install shredder
or from source:
$ sudo python setup.py install
Getting Started
>>> from shredder import Shredder
>>>
>>>
>>> function work_generator:
>>> with open(f) as f:
>>> yield readline(f)
>>>
>>> function worker(shredder, data):
>>> results = do_something_with_data(data)
>>> return results
>>>
>>> function aggregator(results):
>>> dosomethingwitresults
>>>
>>> shredder = Shredder(
>>> work_generator=work_generator,
>>> worker=worker,
>>> aggregator=aggregator,
>>> num_processes=5,
>>> )
>>>
>>> shredder.start()
>>>
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
shredder-0.5.tar.gz
(3.2 kB
view hashes)