Package to monitor and throttle multiple processes or threads.
Project description
mp_throttle
A Python package to monitor and throttle multiprocessing processes.
Use case:
This package can be used to control and monitor Pythons multiprocessing processes or threading threads. The central Throttle class can be used to monitor the current and the mean frequency of processes/threads and/or to throttle them to execute a limited times per second. This can be used to limit the server requests of a multi-process web crawler.
Example:
import multiprocessing
import mp_throttle
import time
def work(tank):
while not tank.kill_flag.is_set():
# Block until 'fuel' is available.
tank.await_fuel()
# do something
return
# Limit the processes to 4 per 1 second, by providing 4 'fuel' per second.
throttle = mp_throttle.Throttle(4,1)
workerpool = multiprocessing.Pool(processes=4, initializer=work, initargs=(throttle,))
throttle.start()
time.sleep(5)
# Stops the processes and return stats:
runtime, total_processes, mean_frequency, mean_processes_per_second = throttle.stop()
print("Runtime: {}".format(runtime))
print("Total: {}".format(total_processes))
print("Mean frequency: {}".format(mean_frequency))
print("Processes per second: {}".format(mean_processes_per_second))
Output:
>>> Runtime: 5.027516841888428
>>> Total: 20
>>> Mean frequency: 0.25046865940093993
>>> Processes per second: 3.9925154803469485
Installation:
To install mp_throttle, simply use pip:
pip install mp_throttle
Documentation:
For the full documentation see docs.elpunkt.eu
How to Contribute:
- Test mp_throttle and open an issue to report a bug or discuss a feature idea.
- Give general feedback on the code and the package structure. Since this is my first python package, I'm sure there is a lot to feedback on:)
- Fork the repository and make your changes.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
mp_throttle-0.0.5.tar.gz
(5.0 kB
view details)
Built Distribution
File details
Details for the file mp_throttle-0.0.5.tar.gz
.
File metadata
- Download URL: mp_throttle-0.0.5.tar.gz
- Upload date:
- Size: 5.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.11.1 setuptools/40.0.0 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.5.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a2ee2de19476d85c52bcfdcb2f8b3e546b69bdfcb7d9262421326b7244c897b |
|
MD5 | da13670d68b06caf1aafce8b27e36add |
|
BLAKE2b-256 | 24b1fe3c125aca3b2bcc9958bbfac7708dcead03de41a936b98f12f44a051d49 |
Provenance
File details
Details for the file mp_throttle-0.0.5-py3-none-any.whl
.
File metadata
- Download URL: mp_throttle-0.0.5-py3-none-any.whl
- Upload date:
- Size: 5.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.11.1 setuptools/40.0.0 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.5.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9cc8946dd73759b318cd39b44ed68b290379ae45a023fe5dfbce774fe16e0e87 |
|
MD5 | de4f98208b03eb93d85a4877fb021274 |
|
BLAKE2b-256 | 79559e98d9e1c6122bbf9126d01747f20ffa53d378d17b1ba2e36c97469ee9f3 |