Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (
Help us improve Python packaging - Donate today!

An implementation of deferToProcess using the multiprocessing package.

Project Description


The Python Global Interpreter Lock (GIL) poses some limitations on multi processor performance by locking on all pure Python code. Although the threading support in twisted is excellent, the GIL was holding back performance by essentially keeping the process bound to a single CPU.

In order to get around the GIL, Python brought in the multiprocessing package as of version 2.6. The multiprocessing package has been backported to both Python 2.4 and 2.5 and is available from the Python Package Index ( The multiprocessing package provides a similar interface to the threading libraries provided by Python except that it uses processes instead of threads.

This package, twisted.internet.processes, implements a wrapper around the multiprocessing library and provides a method, deferToProcess, that works in the same manner as deferToThread.


A process pool is created on the reactor in the same manner that the thread pool is created in the normal reactor when deferToThread is first called.

Initially, the goal was to use the non-blocking code and callbacks from the multiprocessing library to provide a deferToProcess method. Unfortunately, the multiprocessing.Pool.apply_async does not call the callback in the event of a failure. In order to retain failure callbacks, the get method from the asynchronous result set, multiprocessing.pool.AsyncResult, is used by wrapping it in a call to deferToThread. The thread is blocked while waiting on the results from the call in a separate process.

Using the multiprocessing.Pool.apply_async to defer to a process imposes some additional requirements on the user code. All arguments and the function itself must be pickleable by the Python cPickle module.


Copyright 2009 Texas A&M University

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.


1.0b1 2009/08/24

  • Initial release
  • Locked to multiprocessing or below due to a bug in logging in
  • No testing yet
  • deferToProcess based on a blocking call wrapped in deferToThread
Release History

Release History

This version
History Node


Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
twisted.internet.processes-1.0b1.tar.gz (3.8 kB) Copy SHA256 Checksum SHA256 Source Aug 24, 2009

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting