Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Replacement for multiprocessing.Pool.

Project Description

Steam factory

Replacement for multiprocessing’s Pool, offering more powerful features. Allow running a generic Python function asynchronously.

Example usage

First, we are going to need a function that does something. In this case, it does nothing more than waiting one second.

import time

def do_nothing():
    time.sleep(1)
    print('Sleeping done')  # If you want some feedback..

To run this function in parallel, we’re going to need a Factory instance.

from steamfactory import Factory

# Create a factory, running up to 4 tasks concurrently
factory = Factory(size=4)

All set, we can schedule some async function executions:

for _ in range(4):
    factory.run(do_nothing)

After a second, you should see the four “Sleeping done” messages being printed at once.

In case you’re using this inside a script, and you need the main process to wait for all tasks to be executed before terminating (meaning that tasks will be lost), remember to call the shutdown() method:

factory.shutdown()

Getting feedback

How to get “feedback” from the tasks usually greatly depends on the application. Many times you don’t even bother with the function return value, you just need something to be done. Other times values might be large, or the required retention time might vary.

The library doesn’t currently offer any way to return results to the caller, but you can easily do something like this:

import time
from multiprocessing import Manager

from steamfactory import Factory

_mgr = Manager()
results = _mgr.dict()  # Shared between processes


def addup(a, b):
    time.sleep(1)
    results[(a, b)] = a + b

# Create a factory, running up to 4 tasks concurrently
factory = Factory(size=4)

# Let's schedule some tasks
factory.run(addup, 1, 2)
factory.run(addup, 3, 4)
factory.run(addup, 5, 6)
factory.run(addup, 7, 8)

factory.shutdown()

# Now, results contains all the results (after a 1s processing
# time)

Changelog

0.1

Initial release, implementing functionality to run functions asynchronously.

Release History

Release History

This version
History Node

0.1

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
SteamFactory-0.1-py2-none-any.whl (5.7 kB) Copy SHA256 Checksum SHA256 2.7 Wheel Jun 12, 2016
SteamFactory-0.1-py3-none-any.whl (5.7 kB) Copy SHA256 Checksum SHA256 3.5 Wheel Jun 12, 2016
SteamFactory-0.1.tar.gz (3.9 kB) Copy SHA256 Checksum SHA256 Source Jun 12, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting