What started as a simple CLI/Service to evaluate Kafka cluster topics which have no activity, ended up being a somewhat comprehensive way to monitor Kafka cluster activities.
Takes a configuration file as input, where you can list one or multiple cluster(s) you wish to monitor.
After a set period of time, it can produce a report (to local disk or AWS S3) with the list of topics that haven’t seen any activity. It also exposes metrics via a prometheus endpoint.
Usage
kafka-overwatch -c config.local.yamlFeatures
Supports evaluating multiple Kafka clusters at once
Generates a report on topic usage based on topic watermarks offsets (store local or to S3)
Generates commands script to re-create all the topics in case of DR (store local or to S3)
Exposes metrics via prometheus
Topics count
Partitions count
Number of new messages (measured with topic offsets)
AWS Secret integration for client config values
Schema Registry integration
Scan schema registries, map 1 to many kafka clusters
Backup of the schemas, and CLI to restore schemas to existing/new registry.
Upcoming
Multi-nodes awareness (split the load with multiple nodes)
cfn-kafka-admin output format
topic messages meta-data analysis (i.e are messages compressed?)
scripts to perform cleanup
Recommendations generated from/based on models
Conduktor Gateway vClusters auto-discovery
Configuration
Whilst a much more comprehensive documentation is yet to be written, please look at kafka_overwatch/specs/config.json which is used with jsonschema to perform validation of the input.
Return codes
0 - all successful. 1 - error during execution 2 - error importing configuration.
Misc
Thanks
Thanks to the Apache Kafka OpenSource community for their continuous efforts in making the eco-system great. Thanks to the NASA for having a public cluster to run tests with
Note
Inspired by kafka-idle-topics, yet completely re-written to be a continuous monitoring of the topics, similar to cruise-control.
Status
Images build status
Docs build status
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file kafka_overwatch-0.5.0.post0.tar.gz.
File metadata
- Download URL: kafka_overwatch-0.5.0.post0.tar.gz
- Upload date:
- Size: 44.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
poetry/1.8.2 CPython/3.12.2 Linux/6.5.6-300.fc39.x86_64
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
05be02ab6616a43c345ced8796f6ef12580b4a93d23ef8df490393a753ed7e89
|
|
| MD5 |
9630d384e8d482822ce4ed05aa402fb4
|
|
| BLAKE2b-256 |
c405f211b2bf0745b9ec9f48a96ba23c7e17021dcc570c5527882a2661377695
|
File details
Details for the file kafka_overwatch-0.5.0.post0-py3-none-any.whl.
File metadata
- Download URL: kafka_overwatch-0.5.0.post0-py3-none-any.whl
- Upload date:
- Size: 69.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
poetry/1.8.2 CPython/3.12.2 Linux/6.5.6-300.fc39.x86_64
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9df30ff754b541b1c700b34ea4a14cfecfdcfb2b4886bca7f9b78a3ee191268f
|
|
| MD5 |
bcc49c6f1203f6a0004c234861c2c435
|
|
| BLAKE2b-256 |
e712ed40e1661d8e61e3256a45321c2d00c86c5ae327328c3ba8776b522c6503
|
