What are mnqueues?
mnqueues stands for Monitored Queues - a coupling between Python multiprocessing Queue() and a Monitor entity. A Monitor collect and alerts on Queue usage statistics.
Tracked measures
mnqueues tracks several measures per queue:
- Average number of writes to queue per minute,
- Average number of reads to queue per minute,
- Time spend in queue (latency) in milliseconds.
Installation
To install mnqueues type:
pip install mnqueues
How-To use
To create a monitored queue:
import mnqueues as mnq
from mnqueues.gcp_monitor import GCPMonitor
q = mnq.MNQueue(monitor=GCPMonitor("some-unique-name"))
The MNQueue() object encapsulated Python multiprocessing.Queue() and supports same functions. The MNQueue() object can be passed between processes, like a Queue() object.
Monitors
File Logger
from mnqueues.log_monitor import LOGMonitor
monitor = LOGMonitor("log-file-name")
Log all put() and get() calls to a log file with the the name log-file-name.log with the following format:
[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 5003
[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 4997
Google Cloud Monitor (using StackDriver)
from mnqueues.gcp_monitor import GCPMonitor
monitor = GCPMonitor("unique-name")
All calls to put() and get() are sent to Google Cloud Monitor. The Monitor class sends data to two custom measures:
OpenCensus/mnqueues.{name}.number_queue_get(line, no aggregation on GCP required)OpenCensus/mnqueues.{name}.number_queue_put(line, no aggregation on GCP required)OpenCensus/mnqueues.{name}.time_in_queue_distribution(heat-map with sum, shows latency distribution)
Note that {name} is passed as a parameter when constructing the Monitor and it aims to assist in creating dash-boards for specific use-cases.
See for details.
Examples
GCP
- View Google Cloud (GCP) Monitoring dashboard showing queue.put() and queue.get() rates per second, generated by running
pyteston the project tests folder. - Monitoring queues with real-time web-socket trading data for LiuAlgoTrader.
Further examples
Can be found in the tests folder.
Contributing
Contributions are highly appreciated. Please review our
Code of Conduct. Bug reports & feature requests can be left in the Issues section, or email me at amor71@sgeltd.com
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file mnqueues-0.0.39.tar.gz.
File metadata
- Download URL: mnqueues-0.0.39.tar.gz
- Upload date:
- Size: 5.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/4.0.1 CPython/3.9.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1797e333421ceab4b0b98569861a9d50697dd11da2f8e32334f66d30ab4cbaae
|
|
| MD5 |
8c4f7e2e5c0abf6822e0b9ccfedae92a
|
|
| BLAKE2b-256 |
15403a4c357d7ff794ff9e5643aa46d36eb3c4f26db2b055819ca9db9b00b196
|
