What are mnqueues?
mnqueues stands for Monitored Queues - a coupling between Python multiprocessing Queue() and a Monitor entity. A Monitor collect and alerts on Queue usage statistics.
Tracked measures
mnqueues tracks several measures per queue:
- Average number of writes to queue per minute,
- Average number of reads to queue per minute,
- Time spend in queue (latency) in milliseconds.
Installation
To install mnqueues type:
pip install mnqueues
How-To use
To create a monitored queue:
import mnqueues as mnq
from mnqueues.gcp_monitor import GCPMonitor
q = mnq.MNQueue(monitor=GCPMonitor("some-unique-name"))
The MNQueue() object encapsulated Python multiprocessing.Queue() and supports same functions. The MNQueue() object can be passed between processes, like a Queue() object.
Monitors
File Logger
from mnqueues.log_monitor import LOGMonitor
monitor = LOGMonitor("log-file-name")
Log all put() and get() calls to a log file with the the name log-file-name.log with the following format:
[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 5003
[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 4997
Google Cloud Monitor (using StackDriver)
from mnqueues.gcp_monitor import GCPMonitor
monitor = GCPMonitor("unique-name")
All calls to put() and get() are sent to Google Cloud Monitor. The Monitor class sends data to two custom measures:
OpenCensus/mnqueues.{name}.number_queue_get(line, no aggregation on GCP required)OpenCensus/mnqueues.{name}.number_queue_put(line, no aggregation on GCP required)OpenCensus/mnqueues.{name}.time_in_queue_distribution(heat-map with sum, shows latency distribution)
Note that {name} is passed as a parameter when constructing the Monitor and it aims to assist in creating dash-boards for specific use-cases.
See for details.
Examples
GCP
- View Google Cloud (GCP) Monitoring dashboard showing queue.put() and queue.get() rates per second, generated by running
pyteston the project tests folder. - Monitoring queues with real-time web-socket trading data for LiuAlgoTrader.
Further examples
Can be found in the tests folder.
Contributing
Contributions are highly appreciated. Please review our
Code of Conduct. Bug reports & feature requests can be left in the Issues section, or email me at amor71@sgeltd.com
1 maintainer
Metadata
1 maintainer
Release files for mnqueues 0.0.39
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| mnqueues-0.0.39.tar.gz | 5.5 kB | Without provenance | Details |
Release files / mnqueues-0.0.39.tar.gz
| Download URL | mnqueues-0.0.39.tar.gz |
|---|---|
| Size | 5.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1797e333421ceab4b0b98569861a9d50697dd11da2f8e32334f66d30ab4cbaae
|
|
BLAKE2b-256 checksum How to use checksums |
15403a4c357d7ff794ff9e5643aa46d36eb3c4f26db2b055819ca9db9b00b196
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.9.13
|
Security for mnqueues 0.0.39
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
