Skip to main content

Simple pool manager

Project description

poolmanager

Build Status

Simple pool manager implementation for unordered results

poolmanager is compatible with python 2.6, 2.7, 3.6 and 3.7

Usage

ctrl+c will automatically terminate the main process and thus all the associated subprocesses. You'll need to wait for all the functions to finish their current execution before being able to stop all the processes. One can also easily create a callback function to monitor the computation state.

from poolmanager import PoolManager

def add(x):
    return x + 1.5

def callback(counter, result):
    print 'counter: %s' % counter
    print 'result: %s' % result

def main():
    chunk = 2
    iterator = xrange(0, 10)
    pm = PoolManager(numProcs=2, factor=2, store=True)
    pm.imap_unordered(add, iterator, chunk, callback=callback)

if __name__ == '__main__':
    main()

>> counter: 1
>> result: 1.5
>> counter: 2
>> result: 2.5
>> counter: 3
>> result: 5.5
>> counter: 4
>> result: 6.5
>> counter: 5
>> result: 7.5
>> counter: 6
>> result: 8.5
>> counter: 7
>> result: 3.5
>> counter: 8
>> result: 4.5
>> counter: 9
>> result: 9.5
>> counter: 10
>> result: 10.5

print [i for i in iterator]
print pm.results

>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>> [1.5, 2.5, 5.5, 6.5, 7.5, 8.5, 3.5, 4.5, 9.5, 10.5]

Tests

python setup.py test

Publish a new version of the module

Edit $HOME/.pypirc and add (username and password in keepass):

[distutils]
index-servers =
  pypi
  pypitest

[pypi]
repository=https://upload.pypi.org/legacy/
username=iwi***
password=

[pypitest]
repository=https://test.pypi.org/legacy/
username=iwi***
password=

Bump version in setup.py.

Upload the new module to the test repository:

pip install --upgrade setuptools
python setup.py sdist upload -r pypitest
python setup.py bdist_wheel upload -r pypitest

Test local install from test repository.

pip install -i https://test.pypi.org/simple/ poolmanager

If everything is ok, push the new version to the default repository.

python setup.py sdist upload -r pypi
python setup.py bdist_wheel upload -r pypi

Test the newly created module.

Create a RELEASE in github.

CONTRIBUTORS:

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

poolmanager-0.2.1.tar.gz (3.2 kB view details)

Uploaded Source

Built Distribution

poolmanager-0.2.1-py3-none-any.whl (3.1 kB view details)

Uploaded Python 3

File details

Details for the file poolmanager-0.2.1.tar.gz.

File metadata

  • Download URL: poolmanager-0.2.1.tar.gz
  • Upload date:
  • Size: 3.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/50.1.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.2

File hashes

Hashes for poolmanager-0.2.1.tar.gz
Algorithm Hash digest
SHA256 84e12062ad2485c7c612221cbc5019cb989ee25155b7f1630452c32439443739
MD5 d2da930342a2be0c288383d9ca41b431
BLAKE2b-256 c47c8c34ef7e554f15ef47328daa3cd4b54fefb71766bb9e1246294f63d8873e

See more details on using hashes here.

File details

Details for the file poolmanager-0.2.1-py3-none-any.whl.

File metadata

  • Download URL: poolmanager-0.2.1-py3-none-any.whl
  • Upload date:
  • Size: 3.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/50.1.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.2

File hashes

Hashes for poolmanager-0.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 94c3304aa18cda481f5db2680081101f6dd57d90335e1eb5fb5aeac1d54bdc0c
MD5 deea83870779eb6fb5a2c7f5aa027045
BLAKE2b-256 2deed1bd0eede857fb105aacac15c4ce9633b790577501cd5d0511c669413360

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page