Skip to main content

Monitored Multiprocessing Queues in Python

Project description

CircleCI Updates Python 3 Sourcery

What are mnqueues?

mnqueues stands for Monitored Queues - a coupling between Python multiprocessing Queue() and a Monitor entity. A Monitor collect and alerts on Queue usage statistics.

Tracked measures

mnqueues tracks several measures per queue:

  1. Average number of writes to queue per minute,
  2. Average number of reads to queue per minute,
  3. Time spend in queue (latency) in milliseconds.

Installation

To install mnqueues type:

pip install mnqueues

How-To use

To create a monitored queue:

import mnqueues as mnq
from mnqueues.gcp_monitor import GCPMonitor

q = mnq.MNQueue(monitor=GCPMonitor("some-unique-name"))

The MNQueue() object encapsulated Python multiprocessing.Queue() and supports same functions. The MNQueue() object can be passed between processes, like a Queue() object.

Monitors

File Logger

from mnqueues.log_monitor import LOGMonitor

monitor = LOGMonitor("log-file-name")

Log all put() and get() calls to a log file with the the name log-file-name.log with the following format:

[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 5003
[<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 4997

Google Cloud Monitor (using StackDriver)

from mnqueues.gcp_monitor import GCPMonitor

monitor = GCPMonitor("unique-name")

All calls to put() and get() are sent to Google Cloud Monitor. The Monitor class sends data to two custom measures:

  1. OpenCensus/mnqueues.{name}.number_queue_get (line, no aggregation on GCP required)
  2. OpenCensus/mnqueues.{name}.number_queue_put (line, no aggregation on GCP required)
  3. OpenCensus/mnqueues.{name}.time_in_queue_distribution (heat-map with sum, shows latency distribution)

Note that {name} is passed as a parameter when constructing the Monitor and it aims to assist in creating dash-boards for specific use-cases.

See for details.

Examples

GCP

  1. View Google Cloud (GCP) Monitoring dashboard showing queue.put() and queue.get() rates per second, generated by running pytest on the project tests folder.
  2. Monitoring queues with real-time web-socket trading data for LiuAlgoTrader.

Further examples

Can be found in the tests folder.

Contributing

Contributions are highly appreciated. Please review our Code of Conduct. Bug reports & feature requests can be left in the Issues section, or email me at amor71@sgeltd.com

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mnqueues-0.0.39.tar.gz (5.5 kB view details)

Uploaded Source

File details

Details for the file mnqueues-0.0.39.tar.gz.

File metadata

  • Download URL: mnqueues-0.0.39.tar.gz
  • Upload date:
  • Size: 5.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.13

File hashes

Hashes for mnqueues-0.0.39.tar.gz
Algorithm Hash digest
SHA256 1797e333421ceab4b0b98569861a9d50697dd11da2f8e32334f66d30ab4cbaae
MD5 8c4f7e2e5c0abf6822e0b9ccfedae92a
BLAKE2b-256 15403a4c357d7ff794ff9e5643aa46d36eb3c4f26db2b055819ca9db9b00b196

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page