Skip to main content

Asynchronous task queueing with Twisted: threaded, multicore, and remote.

Project description

Asynchronous task queueing based on the Twisted framework, with task prioritization and a powerful worker interface. Worker implementations are included for running tasks asynchronously in the main thread, in separate threads, in separate Python interpreters (multiprocessing), and even on separate devices using Twisted's Asynchronous Message Protocol.

Includes deferred iteration capability: Calling a task that returns an iterator can return a Deferator instead, which does the iteration in a Twisted-friendly fashion, even over a network connection. You can also supply an object conforming to Twisted's IConsumer interface and iterations will be fed to it as they become available.

The util module contains a DeferredTracker object that makes the import worthwhile all on its own. You can use its put method to track Twisted Deferred objects without inserting anything into their callback chains. Then you can wait in non-blocking Twisted fashion for all, any, or some of the tracked deferreds to fire (again, without getting tangled up with any of their callbacks) using the tracker's deferToAll, deferToAny, and deferUntilFewer methods.

Includes an example package mcMandelbrot that generates Mandelbrot set images, row by row, demonstrating the power of asynchronous multi-core processing. An instance of ProcessQueue dispatches the computations for each row of pixels to workers running on separate Python processes. The color-mapped RGB results are collected as they come back and intelligently buffered for iterating in a proper sequence to a third-party PNG library that wouldn't ordinarily play nice with Twisted.

You can try things out after installation by running mcmandelbrot (with a few options and arguments) from the console. The output of the script is a PNG file, which you can view by piping to the free Feh image viewer: Just add |feh - at the end of the command line.

There was some effort toward Python 3 compatiblity a while ago, but it's still not yet supported.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
AsynQueue-0.9.8.tar.gz (136.3 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page