Asynchronous task queueing with Twisted: threaded, multicore, and remote.
Project description
Asynchronous task queueing based on the Twisted framework, with task
prioritization and a powerful worker interface. Worker implementations
are included for running tasks asynchronously in the main thread, in
separate threads, in separate Python interpreters (multiprocessing),
and even on separate devices using Twisted's Asynchronouse Message
Protocol.
Includes deferred iteration capability: Calling a task that returns an
iterator can return a Deferator_ instead, which does the iteration in
a Twisted-friendly fashion, even over a network connection. You can
also supply an object conforming to Twisted's IConsumer interface and
iterations will be fed to it as they become available.
Includes an example package mcMandelbrot_ that generates Mandelbrot
set images, row by row, demonstrating the power of asynchronous
multi-core processing. An instance of ProcessQueue_ dispatches the
computations for each row of pixels to workers running on separate
Python processes. The color-mapped RGB results are collected as they
come back and intelligently buffered for iterating in a proper
sequence to a third-party PNG library that wouldn't ordinarily play
nice with Twisted.
You can try things out after installation by running "mcmandelbrot"
(with a few options and arguments) from the console. The output of the
script is a PNG file, which you can view by piping to the free Feh
image viewer: Just add "|feh -" at the end of the command line.
There was some effort toward Python 3 compatiblity a while ago, but
it's still not yet supported.
.. _mcMandelbrot: http://edsuom.com/mcMandelbrot.html
.. _ProcessQueue: http://edsuom.com/AsynQueue/asynqueue.process.ProcessQueue.html
.. _Deferator: http://edsuom.com/AsynQueue/asynqueue.iteration.Deferator.html
prioritization and a powerful worker interface. Worker implementations
are included for running tasks asynchronously in the main thread, in
separate threads, in separate Python interpreters (multiprocessing),
and even on separate devices using Twisted's Asynchronouse Message
Protocol.
Includes deferred iteration capability: Calling a task that returns an
iterator can return a Deferator_ instead, which does the iteration in
a Twisted-friendly fashion, even over a network connection. You can
also supply an object conforming to Twisted's IConsumer interface and
iterations will be fed to it as they become available.
Includes an example package mcMandelbrot_ that generates Mandelbrot
set images, row by row, demonstrating the power of asynchronous
multi-core processing. An instance of ProcessQueue_ dispatches the
computations for each row of pixels to workers running on separate
Python processes. The color-mapped RGB results are collected as they
come back and intelligently buffered for iterating in a proper
sequence to a third-party PNG library that wouldn't ordinarily play
nice with Twisted.
You can try things out after installation by running "mcmandelbrot"
(with a few options and arguments) from the console. The output of the
script is a PNG file, which you can view by piping to the free Feh
image viewer: Just add "|feh -" at the end of the command line.
There was some effort toward Python 3 compatiblity a while ago, but
it's still not yet supported.
.. _mcMandelbrot: http://edsuom.com/mcMandelbrot.html
.. _ProcessQueue: http://edsuom.com/AsynQueue/asynqueue.process.ProcessQueue.html
.. _Deferator: http://edsuom.com/AsynQueue/asynqueue.iteration.Deferator.html
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
AsynQueue-0.9.6.tar.gz
(134.5 kB
view hashes)