parallel-execute 2.0.3

Python wrappers for easy multiprocessing and threading

parallel-execute

Lightweight Python wrappers for easy multiprocessing and multithreading.

Run multiple functions in parallel using a simple API built on top of threading or multiprocessing. Supports error handling, result tracking, timeouts, and controlled concurrency.

Installation

Install the package using pip:

pip install parallel-execute

Usage Example

1. Create a Loom

A Loom takes multiple tasks (functions) and executes them in parallel using either threads or processes.

Using Threads:

from pexecute.thread import ThreadLoom
loom = ThreadLoom(max_runner_cap=10)

Using Processes:

from pexecute.process import ProcessLoom
loom = ProcessLoom(max_runner_cap=10)

• max_runner_cap: The maximum number of threads/processes to run in parallel. You can queue as many functions as needed; only max_runner_cap will run at the same time.

2. Add Tasks to the Loom

Add a Single Task:

Use add_function to add individual functions:

loom.add_function(f1, args1, kw1)
loom.add_function(f2, args2, kw2)
loom.add_function(f3, args3, kw3)

Add Multiple Tasks at Once:

Use add_work to add a batch of functions:

work = [(f1, args1, kwargs1), (f2, args2, kwargs2), (f3, args3, kwargs3)]
loom.add_work(work)

3. Execute Tasks

Once all tasks are added, call execute() to run them. It returns a dictionary mapping each task to its result.

output = loom.execute()

By default, the keys are integers in the order the functions were added. Each value is a dictionary containing the result and execution metadata.

Example:

def fun1():
    return "Hello World"

def fun2(a):
    return a

def fun3(a, b=0):
    return a + b

loom.add_function(fun1, [], {})
loom.add_function(fun2, [1], {})
loom.add_function(fun3, [1], {'b': 3})

output = loom.execute()

>>> output
{
  0: {
      'output': 'Hello World',
      'got_error': False,
      'error': None,
      'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 395002),
      'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396500),
      'execution_time': 0.001498,
     },
  1: {
      'output': 1,
      'got_error': False,
      'error': None,
      'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396590),
      'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 397651),
      'execution_time': 0.001061
     },
  2: {
      'output': 4,
      'got_error': False,
      'error': None,
      'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 400323),
      'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 401749),
      'execution_time': 0.001426
     }
}

4. Using Custom Keys

You can assign a custom key to each task. This allows you to identify results more explicitly in the output dictionary.

loom.add_function(fun1, [], {}, 'key1')
loom.add_function(fun2, [1], {}, 'fun2')
loom.add_function(fun3, [1], {'b': 3}, 'xyz')

output = loom.execute()

>>> output
{
  'key1': {
      'output': 'Hello World',
      'got_error': False,
      'error': None,
      'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 395002),
      'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396500),
      'execution_time': 0.001498,
     },
  'fun2': {
      'output': 1,
      'got_error': False,
      'error': None,
      'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 396590),
      'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 397651),
      'execution_time': 0.001061
     },
  'xyz': {
      'output': 4,
      'got_error': False,
      'error': None,
      'started_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 400323),
      'finished_time': datetime.datetime(2019, 6, 28, 19, 44, 58, 401749),
      'execution_time': 0.001426
     }
}

Migration Notice

parallel-execute is now powered by a newer, more powerful backend called concurra.

New users are encouraged to switch to the new interface:

from concurra import TaskRunner

runner = TaskRunner()
runner.add_task(my_func, *args, **kwargs)
results = runner.run()

Backward compatibility with ThreadLoom and ProcessLoom is currently maintained, but may be deprecated in future versions.