No project description provided
Project description
Kafka group lag aggregate monitor
This utility can currently
- Accept
kafka-consumer-groups
output like file from stdin and print aggregated output to stdout. - ssh to a remote machine with kafka running on it, run
kafka-consumer-groups
, for multiple groups, collect the output, group by group and topic and finally print average and max lag.
Currently Tested on: Python 3.11.3 and macos
Installation
pip install kafka-lag-monitor
Run
stdin-mode
Run the below to get a general idea of how the output would look like.
cat examples/example1.txt | kafka-lag-monitor stdin-mode
- Add option -v to get verbose output
- Try option --tablefmt psql to get tabular output just like psql. (Can try any format supported here)
remote-mode
kafka-lag-monitor -v --remote ubuntu@127.0.0.1 -i ~/.ssh/key.pem --bootstrap-server 127.0.0.1:9000 --group group1 --group group2
Setup locally
Install dependancies
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip3 install poetry
poetry install
TODO
Explore using tui which can monitor lag live by recording last n entries and calculating lag increase/decrease.Live TUI Created. lag increase/decrease pending.- Currently stdin-mode supports only one file? Look for ways to concat multiple outputs and aggregate?
- Test cases
- monitor locally instead of remote? (use sdk or subprocess calls)
- fake lag generator for testing
- Currently refresh interval is hardcoded. (It breaks if the refresh of data taks longer than interval. FIX IT)
- re-use ssh connection across calls in tui-mode. (Might need major refactor)
- Make it look better. (hello darkness (css/tcss) my old friend)
- Ability to toggle select/deselect of kafka groups during watch mode.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for kafka_lag_monitor-0.1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc8a4cc554df5ca82e45a199a1f797a08cb0fc662d4955bb01b5886d5575dd7c |
|
MD5 | 10be0fbce95047bba1ae5765c53c4d4e |
|
BLAKE2b-256 | b3f8f39d99ce47c2840b0898dc6faa847f2bd4429216cb6f8848a1508009e0fc |