Skip to main content

Simple interface for numpy arrays in multiprocessing contexts.

Project description

contextshare

Simple context manager to share numpy arrays in python multiprocessing. Installation on Python >= 3.8:

pip install contextshare

Example

The following code example is complete and compares a serial() implementation with the parallel() version thereof implemented with contextshare.

#!/usr/bin/env python
import contextshare as shared
import numpy as np


def serial(sigmas):
    alphas = []
    for sigma in sigmas:
        K = np.exp(-0.5 * D / (sigma * sigma))
        alphas.append(np.linalg.solve(K, b))
    return alphas


def parallel(sigmas):
    with shared.SharedMemory({"D": D, "b": b}, nworkers=11) as sm:

        @sm.register
        def one_sigma(sigma):
            K = np.exp(-0.5 * D / (sigma * sigma))
            return np.linalg.solve(K, b)

        for sigma in sigmas:
            one_sigma(sigma)

        return sm.evaluate(progress=True)


if __name__ == "__main__":
    N = 2000
    D = np.random.random((N, N))
    b = np.random.random(N)

    sigmas = 2.0 ** np.arange(1, 12)

    s = serial(sigmas)
    p = parallel(sigmas)

Documentation

The context manager shared.SharedMemory takes two arguments: the dictionary of numpy arrays to make available on all parallel workers and the number of workers to create.

# makes variables D and b available under the same name on 11 workers
with shared.SharedMemory({"D": D, "b": b}, nworkers=11) as sm:

The workers are spawned upon entering the context and are stopped upon exiting. Shared memory references are cleaned up automatically. Note that only numpy arrays are supported for sharing. Other arguments should be placed in the arguments of the function below. The function to call (i.e. the body of the serial for loop) needs to be placed in a function and either decorated or called explicitly:

# decorator
@sm.register
def one_sigma(sigma):
    pass

# or, equivalently, an explicit call
sm.register(one_sigma)        

Calling this decorated function returns immediately and enqueues a function call. Calling

sm.evaluate(progress=True)

starts the calculations and returns the results in order. With progress=True a progress bar is shown, default is to be silent.

Credits

This package was created with Cookiecutter using the audreyr/cookiecutter-pypackage` template.

======= History

0.1.0 (2021-02-23)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

contextshare-0.1.1.tar.gz (10.3 kB view details)

Uploaded Source

Built Distribution

contextshare-0.1.1-py3-none-any.whl (4.5 kB view details)

Uploaded Python 3

File details

Details for the file contextshare-0.1.1.tar.gz.

File metadata

  • Download URL: contextshare-0.1.1.tar.gz
  • Upload date:
  • Size: 10.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/45.2.0.post20200210 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.6.10

File hashes

Hashes for contextshare-0.1.1.tar.gz
Algorithm Hash digest
SHA256 cc4fc581ad15838cce66530a6ab01241ac500e7e82a638b5ce6cfbb44562dffa
MD5 7a8be58a228e7232087e967625bf3d6d
BLAKE2b-256 95c1ff49210997caacc316ff47a70e1277f5bec1fe17c2b559039515cfa6c5b5

See more details on using hashes here.

File details

Details for the file contextshare-0.1.1-py3-none-any.whl.

File metadata

  • Download URL: contextshare-0.1.1-py3-none-any.whl
  • Upload date:
  • Size: 4.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/45.2.0.post20200210 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.6.10

File hashes

Hashes for contextshare-0.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 f47bfaa7aff5212afdbcfa472e227e0045d40284d64f9318c9cd5cbababf8df1
MD5 c12ad24e99899bdea8a92f18b05fbeb9
BLAKE2b-256 186fed2e1dd3e900b5573546ed2d482c32a8084a3e61582508ae512e37113137

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page