Skip to main content

Dispatch your trivially parallizable jobs with sharedmem.

Project description

Dispatch your trivially parallizable jobs with sharedmem.

Now also supports Python 3.

  • sharedmem.empty creates numpy arrays to child processes.
  • sharedmem.MapReduce dispatches work to child processes.
  • sharedmem.MapReduce.ordered and sharedmem.MapReduce.critical provides the equivlant concept of OpenMP ordered and OpenMP critical sections.

Functions and variables are inherited from a fork and copy-on-write. Pickability is not a concern.

Easier to use than multiprocessing.Pool, at the cost of not supporting Windows.

For documentation, please refer to http://rainwoodman.github.io/sharedmem .

>>>
>>> input = numpy.arange(1024 * 1024 * 128, dtype='f8')
>>> output = sharedmem.empty(1024 * 1024 * 128, dtype='f8')
>>> with MapReduce() as pool:
>>>    chunksize = 1024 * 1024
>>>    def work(i):
>>>        s = slice (i, i + chunksize)
>>>        output[s] = input[s]
>>>        return i, sum(input[s])
>>>    def reduce(i, r):
>>>        print('chunk', i, 'done')
>>>        return r
>>>    r = pool.map(work, range(0, len(input), chunksize), reduce=reduce)
>>> print numpy.sum(r)
>>>

Project details


Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page