Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Multiprocessing Gevent Extension

Project description

Warning

HIC SUNT DRACONES!!!

This code is extremely experimental (pre-pre-alpha). There is very little testing, lots of things are in flux, some platforms don’t work at all.

DO NOT use in production. This code may crash your server, bankrupt your company, burn your house down and be mean to your puppy. You’ve been warned.

Problem

Due to internal implementation, multiprocessing (MP) is unsafe to use with Gevent even when monkey-patched. Namely, the use of OS semaphore primitives and inter-process IO in MP will cause the main loop to stall/deadlock/block (specific issue depends on the version of CPython).

Solution

Geventmp (Gee-vent Em-Pee, not Gee-ven Tee-Em-Pee) is an extension plugin for monkey patch subsystem of Gevent. As with the rest of the monkey patch subsystem the process is fairly clear:

  1. Identify all places where blocking occurs and where it may stall the loop.
  2. If blocking occurs on a file descriptor (FD), try to convert the file descriptor from blocking to non-blocking (sockets/pipes/fifos, sometimes even files where, rarely, appropriate) and replace blocking IO functions with their gevent non-blocking equivalents.
  3. If blocking occurs in a Python/OS primitive that does not support non-blocking access and thus cannot be geventized, wrap all blocking access to that primitive with native thread-pool-based wrappers and call it a day (while fully understanding that primitive access latency will increase and raw performance may suffer as a result).
  4. If you are really brave and have lots of free time on your hands, completely replace a standard blocking Python non-FD-based primitive with implementation based on an FD-based OS primitive (e.g. POSIX semaphore => Linux eventfd-based semaphore for kernels > 2.6.30).
  5. Due to launching of separate processes in MP, figure out how, when, and whether to monkey patch spawned/forked children and grandchildren.

Installation

The package is hosted on PyPi.

For stable version:

pip install geventmp

For unstable version:

pip install --pre geventmp

Once installed, geventmp will activate by default in the below stanza.

from gevent.monkey import patch_all
patch_all()

If you would like geventmp to not activate by default, either do not install it or explicitly disable it:

from gevent.monkey import patch_all
patch_all(geventmp=False)

That’s it - there are no other flags, settings, properties or config values so far.

Supported Platforms

Note

All claims of support may not be real at all. You’re welcome to experiment. See warnings on top.

  • Linux, possibly Darwin.
  • CPython 2.7, 3.5, 3.6, 3.7… maybe.

TODO

  1. Implement CI/CD once we figure out how Gevent Issue #1448 is going to go.
  2. Test on CPython 2.7, 3.5, 3.6, 3.7, 3.8, PyPys etc.
  3. Monkey patch Windows to the extent possible.
  4. Test and fix Darwin if required.
  5. Lots of applications use Billiard for multiprocessing instead of stock Python package. Consider monkey patching Billiard if detected.

Contact Us

Post feedback and issues on the Bug Tracker, Gitter, and Twitter (@karelleninc).

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for geventmp, version 0.0.1.dev3
Filename, size File type Python version Upload date Hashes
Filename, size geventmp-0.0.1.dev3-py2.py3-none-any.whl (15.4 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page