Skip to main content

Dispatch your trivially parallizable jobs with sharedmem.

Project description

Dispatch your trivially parallizable jobs with sharedmem.

Now also supports Python 3.

  • sharedmem.empty creates numpy arrays to child processes.

  • sharedmem.MapReduce dispatches work to child processes.

  • sharedmem.MapReduce.ordered and sharedmem.MapReduce.critical provides the equivlant concept of OpenMP ordered and OpenMP critical sections.

Functions and variables are inherited from a fork and copy-on-write. Pickability is not a concern.

Easier to use than multiprocessing.Pool, at the cost of not supporting Windows.

For documentation, please refer to .

>>> input = numpy.arange(1024 * 1024 * 128, dtype='f8')
>>> output = sharedmem.empty(1024 * 1024 * 128, dtype='f8')
>>> with MapReduce() as pool:
>>>    chunksize = 1024 * 1024
>>>    def work(i):
>>>        s = slice (i, i + chunksize)
>>>        output[s] = input[s]
>>>        return i, sum(input[s])
>>>    def reduce(i, r):
>>>        print('chunk', i, 'done')
>>>        return r
>>>    r =, range(0, len(input), chunksize), reduce=reduce)
>>> print numpy.sum(r)

Project details

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page