Skip to main content

No project description provided

Project description

Kafka group lag aggregate monitor

This utility can currently

  1. Accept kafka-consumer-groups output like file from stdin and print aggregated output to stdout.
  2. ssh to a remote machine with kafka running on it, run kafka-consumer-groups, for multiple groups, collect the output, group by group and topic and finally print average and max lag.

Currently Tested on: Python 3.11.3 and macos

Installation

pip install kafka-lag-monitor

Run

stdin-mode

Run the below to get a general idea of how the output would look like.

cat examples/example1.txt | kafka-lag-monitor stdin-mode
  1. Add option -v to get verbose output
  2. Try option --tablefmt psql to get tabular output just like psql. (Can try any format supported here)

remote-mode

kafka-lag-monitor -v --remote ubuntu@127.0.0.1 -i ~/.ssh/key.pem --bootstrap-server 127.0.0.1:9000 --group group1 --group group2

Setup locally

Install dependancies

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip3 install poetry
poetry install

TODO

  1. Explore using tui which can monitor lag live by recording last n entries and calculating lag increase/decrease. Live TUI Created. lag increase/decrease pending.
  2. Currently stdin-mode supports only one file? Look for ways to concat multiple outputs and aggregate?
  3. Test cases
  4. monitor locally instead of remote? (use sdk or subprocess calls)
  5. fake lag generator for testing
  6. Currently refresh interval is hardcoded. (It breaks if the refresh of data taks longer than interval. FIX IT)
  7. re-use ssh connection across calls in tui-mode. (Might need major refactor)
  8. Make it look better. (hello darkness (css/tcss) my old friend)
  9. Ability to toggle select/deselect of kafka groups during watch mode.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kafka_lag_monitor-0.1.4.tar.gz (6.0 kB view details)

Uploaded Source

Built Distribution

kafka_lag_monitor-0.1.4-py3-none-any.whl (8.0 kB view details)

Uploaded Python 3

File details

Details for the file kafka_lag_monitor-0.1.4.tar.gz.

File metadata

  • Download URL: kafka_lag_monitor-0.1.4.tar.gz
  • Upload date:
  • Size: 6.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.4.2 CPython/3.11.3 Darwin/23.2.0

File hashes

Hashes for kafka_lag_monitor-0.1.4.tar.gz
Algorithm Hash digest
SHA256 04114ccbce685da4c329e63cf04360e62a4beb363553395ba191b0cffcef2ee1
MD5 e73938fab77fff18df818fb55c28a503
BLAKE2b-256 21e7876fd38ca461c2342b5bde2310289648a7add2b7144650087329fd5f3e51

See more details on using hashes here.

File details

Details for the file kafka_lag_monitor-0.1.4-py3-none-any.whl.

File metadata

File hashes

Hashes for kafka_lag_monitor-0.1.4-py3-none-any.whl
Algorithm Hash digest
SHA256 cc8a4cc554df5ca82e45a199a1f797a08cb0fc662d4955bb01b5886d5575dd7c
MD5 10be0fbce95047bba1ae5765c53c4d4e
BLAKE2b-256 b3f8f39d99ce47c2840b0898dc6faa847f2bd4429216cb6f8848a1508009e0fc

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page