Skip to main content

Python wrappers for easy multiprocessing and threading

Project description

Python wrappers for easy multiprocessing and threading.

Run multiple functions in parallel using parallel-execute

GitHub Latest documentation PyPI PyPI - Wheel PyPI - Python Version

Installation

pip install parallel-execute

Usage Example

1. Create a loom:

This takes a number of tasks and executes them using a pool of threads/process.

  • To use threading
from pexecute.thread import ThreadLoom
loom = ThreadLoom(max_runner_cap=10)
  • To use multiprocessing
from pexecute.process import ProcessLoom
loom = ProcessLoom(max_runner_cap=10)

max_runner_cap: is the number of maximum threads/processes to run at a time. You can add as many as functions you want, but only n functions will run at a time in parallel, n is the max_runner_cap

2. Add tasks in loom

  • Add a function in loom using add_function
loom.add_function(f1, args1, kw1)
loom.add_function(f2, args2, kw2)
loom.add_function(f3, args3, kw3)
  • Add multiple functions together using add_work method
work = [(f1, args1, kwargs1), (f2, args2, kwargs2), (f3, args3, kwargs3)]
loom.add_work(work)

3. Execute all tasks

After adding tasks, calling execute will return a dictionary of results corresponding to the keys or the order in which the tasks were added.

output = loom.execute()

key is the order in which the function was added and value is the return data of the function.

# Example:

def fun1():
   return "Hello World"

def fun2(a):
   return a

def fun3(a, b=0):
   return a+b

loom.add_function(fun1, [], {})
loom.add_function(fun2, [1], {})
loom.add_function(fun3, [1], {'b': 3})

output = loom.execute()
>>> output
    {
     0: {'output': 'Hello World',
         'got_error': False,
         'error': None,
         'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 395002),
         'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396500),
         'execution_time': 0.001498,
         },
     1: {'output': 1,
         'got_error': False,
         'error': None,
         'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396590),
         'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 397651),
         'execution_time': 0.001061
         },
     2: {'output': 4,
         'got_error': False,
         'error': None,
         'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 400323),
         'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 401749),
         'execution_time': 0.001426
         }
    }

We can also provide a key to store the function return data.

# Example:
loom.add_function(fun1, [], {}, 'key1')
loom.add_function(fun2, [1], {}, 'fun2')
loom.add_function(fun3, [1], {'b': 3}, 'xyz')

output = loom.execute()
>>> output
    {
     'key1': {'output': 'Hello World',
             'got_error': False,
             'error': None,
             'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 395002),
             'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396500),
             'execution_time': 0.001498,
             },
     'fun2: {'output': 1,
             'got_error': False,
             'error': None,
             'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396590),
             'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 397651),
             'execution_time': 0.001061
             },
     'xyz': {'output': 4,
             'got_error': False,
             'error': None,
             'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 400323),
             'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 401749),
             'execution_time': 0.001426
             }
    }

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for parallel-execute, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size parallel_execute-0.1.0-py3-none-any.whl (9.3 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size parallel-execute-0.1.0.tar.gz (5.7 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page