This package is a watcher for multiprocessing queues and processes, and contains a series of handy methods to deal with parallel execution.
Runner
The multiwatch.run(processes, queues, exit_event, sleep, report_interval, output) function handles the lifespan of the processes. On regular intervals indicated by report_interval (in seconds), it saves to a file the information about the CPU and memory usage, the different processes running, and the number of elements in a queue.
The individual processes defined through the multiwatch.RunnerProcess class can also specify a retry strategy. When they fail, the runner will restart them automatically.
Termination
multiwatch.setup_sigterm(sigterm_event, exit_event) function ensures the processes are terminated on SIGINT and SIGTERM signals.
Any process can also set the exit_event which will propagate to every other processes an instruction to terminate what they are doing.
Reading files
A common pattern I am using in my applications consists of reading lines from stdin, transforming them, and adding them to a queue in order for other processes to process them.
This can be done in one line of code with multiwatch.read_file_into_queue(file, exit_event, queue, transform). It works in a non-blocking way, that is, the application can still be gracefully terminated through SIGINT and SIGTERM signals (or when it sets the exit_event). In a dedicated article, I explain how this is done, and why the common alternatives don’t work.
Contributing
The source code is available. The package is distributed under MIT License.
If you want to have SVN access to the official repository in order to contribute to the project, contact me at arseni.mourzenko@pelicandd.com. If you find it more convinient to clone the source to GitHub, you can do that too.
1 maintainer
Metadata
1 maintainer
Release files for multiwatch 1.0.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| multiwatch-1.0.7-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Release files / multiwatch-1.0.7-py2.py3-none-any.whl
| Download URL | multiwatch-1.0.7-py2.py3-none-any.whl |
|---|---|
| Size | 7.5 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
9f3e2a5c679aebd6aefbbbec95cd40613e517cfa38373242189c25ae2d068681
|
|
BLAKE2b-256 checksum How to use checksums |
f640c773cd79d8a5bfe36172b9f2308aa325b84f0675f0606d133a073f610027
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/42.0.2 requests-toolbelt/0.8.0 tqdm/4.15.0 CPython/3.6.9
|
Security for multiwatch 1.0.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
