No project description provided
Project description
Autothread
Parallelization made easy.
Autothread allows you to add multithreading/multiprocessing to your functions by adding just a single line:
import autothread
import time
from time import sleep as heavyworkload
@autothread.multithreaded() # <-- This is all you need to add
def example(x: int, y: int):
heavyworkload(1)
return x*y
@autothread.multiprocessed() # <-- Or to use multiprocessing
def example2(x: int, y: int):
heavyworkload(1)
return x*y
Now, instead of integers, your function can take lists of integers. The function will be repeated or each item in your list on a separate thread/process:
start = time.time()
result = example([1, 2, 3, 4, 5], 10)
print(result)
print("Time expired: ", time.time()-start)
>>> [10, 20, 30, 40, 50]
Time expired: 1.0041766166687012
Installing
You can install autothread using:
pip install autothread
Or by cloning the source:
git clone https://github.com/Basdbruijne/autothread.git
cd autothread
pip install -e .
Usage
The autothread.multithreaded
and autothread.multiprocessed
decorators can be placed
in front of any function to make them threaded/multiprocessed. The only requirement
from the function (besides the regular threading and multiprocessing requirements) is
that the function has type-hinting for all the variables that you wish to vary for each thread.
The decorators take 4 arguments to configure the execution:
n_workers
(int): Total number of workers to run in parallel (-1 for unlimited,None
(default) for the amount of cores).mb_mem
(int): Minimum megabytes of memory for each worker, usefull when your script is memory limited.workers_per_core
(int): Number of workers to run per core.progress_bar
(int): Visualize how many of the tasks have started running.
How it works
Autothread uses the type-hinting of your funtion to reliably determine which paremeters you intend to keep constant and which parameters need to change for every thread.
In general, the argument is split over multiple threads/processes when:
- The argument is a list of tuple.
- The argument does not match the type hinting of the function.
- All the items in the argument do match the type hinting of the function.
For example
example(x = [1, 2, 3], y = 5)
x
is a List[int]
, but its original type hint is int
. This means that x
will be split over multiple processes while y=5
stays constant.
example(x = [1, 2, 3], y = [4, 5, 6])
Now, both x
and y
are List[int]
while the original type hint is int
. This means that both
x
and y
will be split over the multiple processes. This requires x
and y
to be of the same
length.
If autothread can't determine the original type hint (e.g. the type hint is missing, incorrect, or
the parameter is part of *args
or **kwargs
), autothread will not divide the list over multiple processes. To override the autodetection of looping parameters for these cases, provide the
_loop_params
keyword with a list of parameters you intend to change for each process when calling your function.
For an overview of more detailed behavior, check threadpy/test.py
. This file contains all the functional tests of autothread.
Error handling
If one of the processes fails, autothread will send a keyboard interrupt signal to all
the other running threads/processes to give them a change to handle the exit gracefully.
If you want the threads to clean things up before exiting, just intercept the KeyboardInterrupt
exeption and do the cleanup (just like you would in a single threaded case).
Autothread itself will exit by raising the exception of the first thread/process that failed.
Known issues
None at the moment, please open a bug if you run into an issue.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.