This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description


The Python Global Interpreter Lock (GIL) poses some limitations on multi processor performance by locking on all pure Python code. Although the threading support in twisted is excellent, the GIL was holding back performance by essentially keeping the process bound to a single CPU.

In order to get around the GIL, Python brought in the multiprocessing package as of version 2.6. The multiprocessing package has been backported to both Python 2.4 and 2.5 and is available from the Python Package Index ( The multiprocessing package provides a similar interface to the threading libraries provided by Python except that it uses processes instead of threads.

This package, twisted.internet.processes, implements a wrapper around the multiprocessing library and provides a method, deferToProcess, that works in the same manner as deferToThread.


A process pool is created on the reactor in the same manner that the thread pool is created in the normal reactor when deferToThread is first called.

Initially, the goal was to use the non-blocking code and callbacks from the multiprocessing library to provide a deferToProcess method. Unfortunately, the multiprocessing.Pool.apply_async does not call the callback in the event of a failure. In order to retain failure callbacks, the get method from the asynchronous result set, multiprocessing.pool.AsyncResult, is used by wrapping it in a call to deferToThread. The thread is blocked while waiting on the results from the call in a separate process.

Using the multiprocessing.Pool.apply_async to defer to a process imposes some additional requirements on the user code. All arguments and the function itself must be pickleable by the Python cPickle module.


Copyright 2009 Texas A&M University

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.


1.0b1 2009/08/24

  • Initial release
  • Locked to multiprocessing or below due to a bug in logging in
  • No testing yet
  • deferToProcess based on a blocking call wrapped in deferToThread
Release History

Release History


This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
twisted.internet.processes-1.0b1.tar.gz (3.8 kB) Copy SHA256 Checksum SHA256 Source Aug 24, 2009

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting