Skip to main content

A simple multiprocessing function pool

Project description

pytaskpool

A simple multiprocess function pooler which create results generators

pytaskpool use the python multiprocessing module

pytaskpool provide an easy way to execute python functions in a pool of process (not threads) for using all your CPU cores. Functions results can then be then obtained in order or not (via a generator method).

Download and docs:
http://pypi.python.org/pypi/pytaskpool
Source code & Development:
https://github.com/govlog/pytaskpool
Issues:
christopher.amiaud@gmail.com

Installation

pytaskpool is available in the python index package (pip), It can be installed running the following command:

$ pip install pytaskpool

or by git like this:

$ git clone https://github.com/govlog/pytaskpool
$ cd pytaskpool
$ sudo python setup.py install

or even in a zip over http : https://github.com/govlog/pytaskpool/archive/master.zip

Usage

This sample code will simultanously launch 8 functions with differents parameters using a pool of 8 processes. The results returned by all launched functions will be get by the method get_sorted_results(), which is a generator returning the functions results in order.

The excepted execution time should be around 1 second.

Sample code:

import pytaskpool as tp
from time import sleep

def my_func(x):
    sleep(1)
    return [x ** x]

mypool = tp.TaskPool([], 8)

for r in range(8):
    mypool.launch(my_func, r)

unsorted = [r for r in mypool.get_unsorted_results()]
sorted = [r for r in mypool.get_sorted_results()]

print "unsorted :",unsorted
print "sorted   :",sorted

Terminal output should look like this:

$ time python test.py
unsorted : [[1], [4], [1], [3125], [823543], [256], [27], [46656]]
sorted   : [[1], [1], [4], [27], [256], [3125], [46656], [823543]]

real    0m1.031s
user    0m0.858s
sys     0m0.193s

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
pytaskpool-1.1.tar.gz (17.3 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page