better multiprocessing and multithreading in Python

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for mmckerns from gravatar.com mmckerns

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers
Report project as malware

Project description

About Multiprocess

multiprocess is a fork of multiprocessing. multiprocess extends multiprocessing to provide enhanced serialization, using dill. multiprocess leverages multiprocessing to support the spawning of processes using the API of the Python standard library’s threading module. multiprocessing has been distributed as part of the standard library since Python 2.6.

multiprocess is part of pathos, a Python framework for heterogeneous computing. multiprocess is in active development, so any user feedback, bug reports, comments, or suggestions are highly appreciated. A list of issues is located at https://github.com/uqfoundation/multiprocess/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query.

Major Features

multiprocess enables:

  • objects to be transferred between processes using pipes or multi-producer/multi-consumer queues

  • objects to be shared between processes using a server process or (for simple data) shared memory

multiprocess provides:

  • equivalents of all the synchronization primitives in threading

  • a Pool class to facilitate submitting tasks to worker processes

  • enhanced serialization, using dill

Current Release

The latest released version of multiprocess is available from:

https://pypi.org/project/multiprocess

multiprocess is distributed under a 3-clause BSD license, and is a fork of multiprocessing.

Development Version

You can get the latest development version with all the shiny new features at:

https://github.com/uqfoundation

If you have a new contribution, please submit a pull request.

Installation

multiprocess can be installed with pip:

$ pip install multiprocess

For Python 2, a C compiler is required to build the included extension module from source. Python 3 and binary installs do not require a C compiler.

Requirements

multiprocess requires:

  • python (or pypy), >=3.8

  • setuptools, >=42

  • dill, >=0.4.0

Basic Usage

The multiprocess.Process class follows the API of threading.Thread. For example

from multiprocess import Process, Queue

def f(q):
    q.put('hello world')

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=[q])
    p.start()
    print (q.get())
    p.join()

Synchronization primitives like locks, semaphores and conditions are available, for example

>>> from multiprocess import Condition
>>> c = Condition()
>>> print (c)
<Condition(<RLock(None, 0)>), 0>
>>> c.acquire()
True
>>> print (c)
<Condition(<RLock(MainProcess, 1)>), 0>

One can also use a manager to create shared objects either in shared memory or in a server process, for example

>>> from multiprocess import Manager
>>> manager = Manager()
>>> l = manager.list(range(10))
>>> l.reverse()
>>> print (l)
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
>>> print (repr(l))
<Proxy[list] object at 0x00E1B3B0>

Tasks can be offloaded to a pool of worker processes in various ways, for example

>>> from multiprocess import Pool
>>> def f(x): return x*x
...
>>> p = Pool(4)
>>> result = p.map_async(f, range(10))
>>> print (result.get(timeout=1))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

When dill is installed, serialization is extended to most objects, for example

>>> from multiprocess import Pool
>>> p = Pool(4)
>>> print (p.map(lambda x: (lambda y:y**2)(x) + x, xrange(10)))
[0, 2, 6, 12, 20, 30, 42, 56, 72, 90]

More Information

Probably the best way to get started is to look at the documentation at http://multiprocess.rtfd.io. Also see multiprocess.tests for scripts that demonstrate how multiprocess can be used to leverge multiple processes to execute Python in parallel. You can run the test suite with python -m multiprocess.tests. As multiprocess conforms to the multiprocessing interface, the examples and documentation found at http://docs.python.org/library/multiprocessing.html also apply to multiprocess if one will import multiprocessing as multiprocess. See https://github.com/uqfoundation/multiprocess/tree/master/py3.12/examples for a set of examples that demonstrate some basic use cases and benchmarking for running Python code in parallel. Please feel free to submit a ticket on github, or ask a question on stackoverflow (@Mike McKerns). If you would like to share how you use multiprocess in your work, please send an email (to mmckerns at uqfoundation dot org).

Citation

If you use multiprocess to do research that leads to publication, we ask that you acknowledge use of multiprocess by citing the following in your publication:

M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,
"Building a framework for predictive science", Proceedings of
the 10th Python in Science Conference, 2011;
http://arxiv.org/pdf/1202.1056

Michael McKerns and Michael Aivazis,
"pathos: a framework for heterogeneous computing", 2010- ;
https://uqfoundation.github.io/project/pathos

Please see https://uqfoundation.github.io/project/pathos or http://arxiv.org/pdf/1202.1056 for further information.

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for mmckerns from gravatar.com mmckerns

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

This version

0.70.18

0.70.17

0.70.16

0.70.15

0.70.14

0.70.13

0.70.12.2

0.70.12.1

0.70.12

0.70.11.1

0.70.11

0.70.10

0.70.9

0.70.8

0.70.7

0.70.6.1

0.70.6

0.70.5

0.70.4

0.70.3

0.70.2

0.70.1

0.52.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

multiprocess-0.70.18.tar.gz (1.8 MB view details)

Uploaded Source

Built Distributions

multiprocess-0.70.18-py313-none-any.whl (151.9 kB view details)

Uploaded Python 3.13

multiprocess-0.70.18-py312-none-any.whl (150.3 kB view details)

Uploaded Python 3.12

multiprocess-0.70.18-py311-none-any.whl (144.5 kB view details)

Uploaded Python 3.11

multiprocess-0.70.18-py310-none-any.whl (134.9 kB view details)

Uploaded Python 3.10

multiprocess-0.70.18-py39-none-any.whl (133.5 kB view details)

Uploaded Python 3.9

multiprocess-0.70.18-py38-none-any.whl (132.6 kB view details)

Uploaded Python 3.8

multiprocess-0.70.18-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl (144.7 kB view details)

Uploaded PyPy manylinux: glibc 2.28+ x86-64

multiprocess-0.70.18-pp311-pypy311_pp73-macosx_11_0_arm64.whl (144.7 kB view details)

Uploaded PyPy macOS 11.0+ ARM64

multiprocess-0.70.18-pp311-pypy311_pp73-macosx_10_15_x86_64.whl (144.7 kB view details)

Uploaded PyPy macOS 10.15+ x86-64

multiprocess-0.70.18-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl (135.1 kB view details)

Uploaded PyPy manylinux: glibc 2.28+ x86-64

multiprocess-0.70.18-pp310-pypy310_pp73-macosx_11_0_arm64.whl (135.1 kB view details)

Uploaded PyPy macOS 11.0+ ARM64

multiprocess-0.70.18-pp310-pypy310_pp73-macosx_10_15_x86_64.whl (135.1 kB view details)

Uploaded PyPy macOS 10.15+ x86-64

multiprocess-0.70.18-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl (133.5 kB view details)

Uploaded PyPy manylinux: glibc 2.28+ x86-64

multiprocess-0.70.18-pp39-pypy39_pp73-macosx_10_13_x86_64.whl (133.5 kB view details)

Uploaded PyPy macOS 10.13+ x86-64

multiprocess-0.70.18-pp39-pypy39_pp73-macosx_10_13_arm64.whl (133.5 kB view details)

Uploaded PyPy macOS 10.13+ ARM64

multiprocess-0.70.18-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl (132.8 kB view details)

Uploaded PyPy manylinux: glibc 2.28+ x86-64

multiprocess-0.70.18-pp38-pypy38_pp73-macosx_10_9_x86_64.whl (132.8 kB view details)

Uploaded PyPy macOS 10.9+ x86-64

multiprocess-0.70.18-pp38-pypy38_pp73-macosx_10_9_arm64.whl (132.8 kB view details)

Uploaded PyPy macOS 10.9+ ARM64

File details

Details for the file multiprocess-0.70.18.tar.gz.

File metadata

  • Download URL: multiprocess-0.70.18.tar.gz
  • Upload date:
  • Size: 1.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.13.3

File hashes

Hashes for multiprocess-0.70.18.tar.gz
Algorithm Hash digest
SHA256 f9597128e6b3e67b23956da07cf3d2e5cba79e2f4e0fba8d7903636663ec6d0d
MD5 98dd542079627bb8ecf764c7a8cb6ccd
BLAKE2b-256 72fd2ae3826f5be24c6ed87266bc4e59c46ea5b059a103f3d7e7eb76a52aeecb

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-py313-none-any.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-py313-none-any.whl
Algorithm Hash digest
SHA256 871743755f43ef57d7910a38433cfe41319e72be1bbd90b79c7a5ac523eb9334
MD5 a60b542e736bf0a8851b322234be00c9
BLAKE2b-256 ee257d7e78e750bc1aecfaf0efbf826c69a791d2eeaf29cf20cba93ff4cced78

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-py312-none-any.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-py312-none-any.whl
Algorithm Hash digest
SHA256 9b78f8e5024b573730bfb654783a13800c2c0f2dfc0c25e70b40d184d64adaa2
MD5 00ba8ba5fea59ace183e6bf351c54e5d
BLAKE2b-256 bfb65f922792be93b82ec6b5f270bbb1ef031fd0622847070bbcf9da816502cc

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-py311-none-any.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-py311-none-any.whl
Algorithm Hash digest
SHA256 5aa6eef98e691281b3ad923be2832bf1c55dd2c859acd73e5ec53a66aae06a1d
MD5 d711bda0d636e2ab183d862921f6173c
BLAKE2b-256 4b889039f2fed1012ef584751d4ceff9ab4a51e5ae264898f0b7cbf44340a859

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-py310-none-any.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-py310-none-any.whl
Algorithm Hash digest
SHA256 60c194974c31784019c1f459d984e8f33ee48f10fcf42c309ba97b30d9bd53ea
MD5 de4611298d59af73c8b6f2b4ac172bd8
BLAKE2b-256 bad80cba6cf51a1a31f20471fbc823a716170c73012ddc4fb85d706630ed6e8f

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-py39-none-any.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-py39-none-any.whl
Algorithm Hash digest
SHA256 e78ca805a72b1b810c690b6b4cc32579eba34f403094bbbae962b7b5bf9dfcb8
MD5 37e388c5bbb93234bd4e11fe3a240cdb
BLAKE2b-256 6c28dd72947e59a6a8c856448a5e74da6201cb5502ddff644fbc790e4bd40b9a

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-py38-none-any.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-py38-none-any.whl
Algorithm Hash digest
SHA256 dbf705e52a154fe5e90fb17b38f02556169557c2dd8bb084f2e06c2784d8279b
MD5 ff819c3e8f7c0f25c65ad04d2a417702
BLAKE2b-256 3bc3ca84c19bd14cdfc21c388fdcebf08b86a7a470ebc9f5c3c084fc2dbc50f7

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 4d77f8e4bfe6c6e2e661925bbf9aed4d5ade9a1c6502d5dfc10129b9d1141797
MD5 1ea576f711a8350b31ed0a500c53cd0f
BLAKE2b-256 dd74cb8c831e58dc6d5cf450b17c7db87f14294a1df52eb391da948b5e0a0b94

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp311-pypy311_pp73-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp311-pypy311_pp73-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 0929ba95831adb938edbd5fb801ac45e705ecad9d100b3e653946b7716cb6bd3
MD5 1ceae4f077ebd18533d102620f22e724
BLAKE2b-256 17bf87323e79dd0562474fad3373c21c66bc6c3c9963b68eb2a209deb4c8575e

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp311-pypy311_pp73-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp311-pypy311_pp73-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 8b8940ae30139e04b076da6c5b83e9398585ebdf0f2ad3250673fef5b2ff06d6
MD5 8cd228b31092013d09f3b9f0499a3a8e
BLAKE2b-256 554d9af0d1279c84618bcd35bf5fd7e371657358c7b0a523e54a9cffb87461f8

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 6fa1366f994373aaf2d4738b0f56e707caeaa05486e97a7f71ee0853823180c2
MD5 4176c716a415bd7837c4cbbaea9bd002
BLAKE2b-256 c1a35f8d3b9690ea5580bee5868ab7d7e2cfca74b7e826b28192b40aa3881cdc

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp310-pypy310_pp73-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp310-pypy310_pp73-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 06b19433de0d02afe5869aec8931dd5c01d99074664f806c73896b0d9e527213
MD5 553a21159c3c800d08c474264a8e5251
BLAKE2b-256 e503b7b10dbfc17b2b3ce07d4d30b3ba8367d0ed32d6d46cd166e298f161dd46

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp310-pypy310_pp73-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp310-pypy310_pp73-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 25d4012dcaaf66b9e8e955f58482b42910c2ee526d532844d8bcf661bbc604df
MD5 30bd3c3e14af784b3a5f274da125c0f9
BLAKE2b-256 c8f87f9a8f08bf98cea1dfaa181e05cc8bbcb59cecf044b5a9ac3cce39f9c449

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 5f9be0342e597dde86152c10442c5fb6c07994b1c29de441b7a3a08b0e6be2a0
MD5 afbe7ad71afe11a09b1cfc2be43cb08d
BLAKE2b-256 7d5a6f3ebcbc1508aa651cbe8deeca612b7915b97303410c93e9a4d83ba07e03

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp39-pypy39_pp73-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp39-pypy39_pp73-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 3fbba48bfcd932747c33f0b152b26207c4e0840c35cab359afaff7a8672b1031
MD5 c7bb8e90df258f6e06323459717eb20e
BLAKE2b-256 4165d74a7955593c2c56c79d01ef07be5aa103342441673657dc7387855703de

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp39-pypy39_pp73-macosx_10_13_arm64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp39-pypy39_pp73-macosx_10_13_arm64.whl
Algorithm Hash digest
SHA256 9fd8d662f7524a95a1be7cbea271f0b33089fe792baabec17d93103d368907da
MD5 89bb6e7ef2e526f5bfd7dce762c8ae70
BLAKE2b-256 1289733ebfc487a4381590d863222f3f0c8bac105d032102d7a96d6566823d67

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 c0c7cd75d0987ab6166d64e654787c781dbacbcbcaaede4c1ffe664720b3e14b
MD5 b0d44cf3658983f9f7fb47ad5c46679a
BLAKE2b-256 3017cf4fb6957efb9e6665134a819809c408afeee9654e8b0f0a8b5de5d5d6ad

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp38-pypy38_pp73-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp38-pypy38_pp73-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 bcac5a4e81f1554d98d1bba963eeb1bd24966432f04fcbd29b6e1a16251ad712
MD5 1198694de787072a95014e526f66e0e3
BLAKE2b-256 b97d934d6411a16870a46d4677f8d7013c536abd41ed8c25897ff872a26b918d

See more details on using hashes here.

File details

Details for the file multiprocess-0.70.18-pp38-pypy38_pp73-macosx_10_9_arm64.whl.

File metadata

File hashes

Hashes for multiprocess-0.70.18-pp38-pypy38_pp73-macosx_10_9_arm64.whl
Algorithm Hash digest
SHA256 2dbaae9bffa1fb2d58077c0044ffe87a8c8974e90fcf778cdf90e139c970d42a
MD5 e4f0c4c887aed00cd132a7659f30f54c
BLAKE2b-256 2d8a625c4aaf84f1654dfa233494accd83723e244ec0d4d26da4642233defc91

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page