twisted.internet.processes 1.0b1

An implementation of deferToProcess using the multiprocessing package.

Introduction

The Python Global Interpreter Lock (GIL) poses some limitations on multi processor performance by locking on all pure Python code. Although the threading support in twisted is excellent, the GIL was holding back performance by essentially keeping the process bound to a single CPU.

In order to get around the GIL, Python brought in the multiprocessing package as of version 2.6. The multiprocessing package has been backported to both Python 2.4 and 2.5 and is available from the Python Package Index (http://pypi.python.org/pypi). The multiprocessing package provides a similar interface to the threading libraries provided by Python except that it uses processes instead of threads.

This package, twisted.internet.processes, implements a wrapper around the multiprocessing library and provides a method, deferToProcess, that works in the same manner as deferToThread.

Implementation

A process pool is created on the reactor in the same manner that the thread pool is created in the normal reactor when deferToThread is first called.

Initially, the goal was to use the non-blocking code and callbacks from the multiprocessing library to provide a deferToProcess method. Unfortunately, the multiprocessing.Pool.apply_async does not call the callback in the event of a failure. In order to retain failure callbacks, the get method from the asynchronous result set, multiprocessing.pool.AsyncResult, is used by wrapping it in a call to deferToThread. The thread is blocked while waiting on the results from the call in a separate process.

Using the multiprocessing.Pool.apply_async to defer to a process imposes some additional requirements on the user code. All arguments and the function itself must be pickleable by the Python cPickle module.

License

Copyright 2009 Texas A&M University

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Changelog

1.0b1 2009/08/24

• Initial release

• Locked to multiprocessing 2.6.1.1 or below due to a bug in logging in 2.6.2.1

• No testing yet

• deferToProcess based on a blocking call wrapped in deferToThread