Skip to main content

Replacement for multiprocessing.Pool.

Project description

Steam factory

Replacement for multiprocessing’s Pool, offering more powerful features. Allow running a generic Python function asynchronously.

Example usage

First, we are going to need a function that does something. In this case, it does nothing more than waiting one second.

import time

def do_nothing():
    time.sleep(1)
    print('Sleeping done')  # If you want some feedback..

To run this function in parallel, we’re going to need a Factory instance.

from steamfactory import Factory

# Create a factory, running up to 4 tasks concurrently
factory = Factory(size=4)

All set, we can schedule some async function executions:

for _ in range(4):
    factory.run(do_nothing)

After a second, you should see the four “Sleeping done” messages being printed at once.

In case you’re using this inside a script, and you need the main process to wait for all tasks to be executed before terminating (meaning that tasks will be lost), remember to call the shutdown() method:

factory.shutdown()

Getting feedback

How to get “feedback” from the tasks usually greatly depends on the application. Many times you don’t even bother with the function return value, you just need something to be done. Other times values might be large, or the required retention time might vary.

The library doesn’t currently offer any way to return results to the caller, but you can easily do something like this:

import time
from multiprocessing import Manager

from steamfactory import Factory

_mgr = Manager()
results = _mgr.dict()  # Shared between processes


def addup(a, b):
    time.sleep(1)
    results[(a, b)] = a + b

# Create a factory, running up to 4 tasks concurrently
factory = Factory(size=4)

# Let's schedule some tasks
factory.run(addup, 1, 2)
factory.run(addup, 3, 4)
factory.run(addup, 5, 6)
factory.run(addup, 7, 8)

factory.shutdown()

# Now, results contains all the results (after a 1s processing
# time)

Changelog

0.1

Initial release, implementing functionality to run functions asynchronously.

Project details


Release history Release notifications

This version
History Node

0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
SteamFactory-0.1-py2-none-any.whl (5.7 kB) Copy SHA256 hash SHA256 Wheel 2.7
SteamFactory-0.1-py3-none-any.whl (5.7 kB) Copy SHA256 hash SHA256 Wheel 3.5
SteamFactory-0.1.tar.gz (3.9 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page