Monitored Multiprocessing Queues in Python
Project description
What are mnqueues?
mnqueues
stands for Monitored Queues - a coupling between Python multiprocessing
Queue() and a Monitor entity. A Monitor collect and alerts on Queue usage statistics.
Tracked measures
mnqueues
tracks several measures per queue:
- Average number of writes to queue per minute,
- Average number of reads to queue per minute,
- Time spend in queue (latency) in milliseconds.
Installation
To install mnqueues
type:
pip install mnqueues
How-To use
To create a monitored queue:
import mnqueues as mnq
from mnqueues.gcp_monitor import GCPMonitor
q = mnq.MNQueue(monitor=GCPMonitor("some-unique-name"))
The MNQueue()
object encapsulated Python multiprocessing.Queue()
and supports same functions. The MNQueue() object can be passed between processes, like a Queue() object.
Monitors
File Logger
from mnqueues.log_monitor import LOGMonitor
monitor = LOGMonitor("log-file-name")
Log all put()
and get()
calls to a log file with the the name log-file-name.log
with the following format:
[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 5003
[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 4997
Google Cloud Monitor (using StackDriver)
from mnqueues.gcp_monitor import GCPMonitor
monitor = GCPMonitor("unique-name")
All calls to put()
and get()
are sent to Google Cloud Monitor. The Monitor class sends data to two custom measures:
OpenCensus/mnqueues.{name}.number_queue_get
(line, no aggregation on GCP required)OpenCensus/mnqueues.{name}.number_queue_put
(line, no aggregation on GCP required)OpenCensus/mnqueues.{name}.time_in_queue_distribution
(heat-map with sum, shows latency distribution)
Note that {name}
is passed as a parameter when constructing the Monitor and it aims to assist in creating dash-boards for specific use-cases.
See for details.
Examples
GCP
- View Google Cloud (GCP) Monitoring dashboard showing queue.put() and queue.get() rates per second, generated by running
pytest
on the project tests folder. - Monitoring queues with real-time web-socket trading data for LiuAlgoTrader.
Further examples
Can be found in the tests folder.
Contributing
Contributions are highly appreciated. Please review our
Code of Conduct. Bug reports & feature requests can be left in the Issues
section, or email me at amor71@sgeltd.com
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file mnqueues-0.0.39.tar.gz
.
File metadata
- Download URL: mnqueues-0.0.39.tar.gz
- Upload date:
- Size: 5.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1797e333421ceab4b0b98569861a9d50697dd11da2f8e32334f66d30ab4cbaae |
|
MD5 | 8c4f7e2e5c0abf6822e0b9ccfedae92a |
|
BLAKE2b-256 | 15403a4c357d7ff794ff9e5643aa46d36eb3c4f26db2b055819ca9db9b00b196 |