Skip to main content

An implementation of deferToProcess using the multiprocessing package.

Project description


The Python Global Interpreter Lock (GIL) poses some limitations on multi processor performance by locking on all pure Python code. Although the threading support in twisted is excellent, the GIL was holding back performance by essentially keeping the process bound to a single CPU.

In order to get around the GIL, Python brought in the multiprocessing package as of version 2.6. The multiprocessing package has been backported to both Python 2.4 and 2.5 and is available from the Python Package Index ( The multiprocessing package provides a similar interface to the threading libraries provided by Python except that it uses processes instead of threads.

This package, twisted.internet.processes, implements a wrapper around the multiprocessing library and provides a method, deferToProcess, that works in the same manner as deferToThread.


A process pool is created on the reactor in the same manner that the thread pool is created in the normal reactor when deferToThread is first called.

Initially, the goal was to use the non-blocking code and callbacks from the multiprocessing library to provide a deferToProcess method. Unfortunately, the multiprocessing.Pool.apply_async does not call the callback in the event of a failure. In order to retain failure callbacks, the get method from the asynchronous result set, multiprocessing.pool.AsyncResult, is used by wrapping it in a call to deferToThread. The thread is blocked while waiting on the results from the call in a separate process.

Using the multiprocessing.Pool.apply_async to defer to a process imposes some additional requirements on the user code. All arguments and the function itself must be pickleable by the Python cPickle module.


Copyright 2009 Texas A&M University

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.


1.0b1 2009/08/24

  • Initial release
  • Locked to multiprocessing or below due to a bug in logging in
  • No testing yet
  • deferToProcess based on a blocking call wrapped in deferToThread

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for twisted.internet.processes, version 1.0b1
Filename, size File type Python version Upload date Hashes
Filename, size twisted.internet.processes-1.0b1.tar.gz (3.8 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page