Skip to main content

Multiprocessing Gevent Extension

Project description

Gitter Build Status Coverage Status

GeventMP Version GeventMP Python Versions GeventMP Downloads Per Day GeventMP Downloads Per Week GeventMP Downloads Per Month

Problem

Due to internal implementation, multiprocessing (MP) is unsafe to use with Gevent even when monkey-patched. Namely, the use of OS semaphore primitives and inter-process IO in MP will cause the main loop to stall/deadlock/block (specific issue depends on the version of CPython).

Solution

GeventMP (Gee-vent Em-Pee) is a gevent multiprocessing extension plugin for the monkey-patching subsystem. As with the rest of the monkey patch subsystem the process is fairly clear:

  1. Identify all places where blocking occurs and where it may stall the loop.

  2. If blocking occurs on a file descriptor (FD), try to convert the file descriptor from blocking to non-blocking (sockets/pipes/fifos, sometimes even files where, rarely, appropriate) and replace blocking IO functions with their gevent non-blocking equivalents.

  3. If blocking occurs in a Python/OS primitive that does not support non-blocking access and thus cannot be geventized, wrap all blocking access to that primitive with native thread-pool-based wrappers and call it a day (while fully understanding that primitive access latency will increase and raw performance may suffer as a result).

  4. If you are really brave and have lots of free time on your hands, completely replace a standard blocking Python non-FD-based primitive with implementation based on an FD-based OS primitive (e.g. POSIX semaphore => Linux eventfd-based semaphore for kernels > 2.6.30).

  5. Due to launching of separate processes in MP, figure out how, when, and whether to monkey-patch spawned/forked children and grandchildren.

Installation

The package is hosted on PyPi.

For stable version:

pip install geventmp

For unstable version:

pip install --pre geventmp

Once installed, GeventMP will activate by default in the below stanza.

from gevent.monkey import patch_all
patch_all()

If you would like GeventMP to not activate by default, either do not install it or explicitly disable it:

from gevent.monkey import patch_all
patch_all(geventmp=False)

That’s it - there are no other flags, settings, properties or config values so far.

Supported Platforms

  • Linux and Darwin.

  • CPython 3.9, 3.10, 3.11, 3.12, 3.13

  • PyPy 3.9, 3.10

Known Issues

  • Multiprocessing forkserver works in GeventMP, but the spawned child isn’t green.

TODO

  1. Monkey patch Windows to the extent possible.

  2. Lots of applications use Billiard for multiprocessing instead of stock Python package. Consider monkey patching Billiard if detected.

Contact Us

Post feedback and issues on the Bug Tracker, Gitter, and Twitter (@karelleninc).

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

geventmp-0.0.9.tar.gz (15.8 kB view details)

Uploaded Source

Built Distribution

geventmp-0.0.9-py3-none-any.whl (20.9 kB view details)

Uploaded Python 3

File details

Details for the file geventmp-0.0.9.tar.gz.

File metadata

  • Download URL: geventmp-0.0.9.tar.gz
  • Upload date:
  • Size: 15.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.0.1 CPython/3.12.8

File hashes

Hashes for geventmp-0.0.9.tar.gz
Algorithm Hash digest
SHA256 09b471ed18438eace5f3e286338de53e50aec1e635bf878bdfd28bf1fc2decfd
MD5 71f741249acee5c9203ea720f5879b1c
BLAKE2b-256 73951701cc8f39117841d36f7f40acfd9c1014e7365995e522c8dd85749dd27f

See more details on using hashes here.

File details

Details for the file geventmp-0.0.9-py3-none-any.whl.

File metadata

  • Download URL: geventmp-0.0.9-py3-none-any.whl
  • Upload date:
  • Size: 20.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.0.1 CPython/3.12.8

File hashes

Hashes for geventmp-0.0.9-py3-none-any.whl
Algorithm Hash digest
SHA256 8a320408c130c325de935853010174f8a6fc7b25506e7f19a514761d8b7ef6c1
MD5 0c7babae5b49ac6756317f62df520244
BLAKE2b-256 b8f6811259ed306a00dcd10cc8097949bf6c4c76c48066ef48ba651dbbe18221

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page