What started as a simple CLI/Service to evaluate Kafka cluster topics which have no activity, ended up being a somewhat comprehensive way to monitor Kafka cluster activities.
Takes a configuration file as input, where you can list one or multiple cluster(s) you wish to monitor.
After a set period of time, it can produce a report (to local disk or AWS S3) with the list of topics that haven’t seen any activity. It also exposes metrics via a prometheus endpoint.
Usage
kafka-overwatch -c config.local.yamlFeatures
Supports evaluating multiple Kafka clusters at once
Generates a report on topic usage based on topic watermarks offsets (store local or to S3)
Generates commands script to re-create all the topics in case of DR (store local or to S3)
Exposes metrics via prometheus
Topics count
Partitions count
Number of new messages (measured with topic offsets)
AWS Secret integration for client config values
Schema Registry integration
Scan schema registries, map 1 to many kafka clusters
Backup of the schemas, and CLI to restore schemas to existing/new registry.
Upcoming
Multi-nodes awareness (split the load with multiple nodes)
cfn-kafka-admin output format
topic messages meta-data analysis (i.e are messages compressed?)
scripts to perform cleanup
Recommendations generated from/based on models
Conduktor Gateway vClusters auto-discovery
Configuration
Whilst a much more comprehensive documentation is yet to be written, please look at kafka_overwatch/specs/config.json which is used with jsonschema to perform validation of the input.
Return codes
0 - all successful. 1 - error during execution 2 - error importing configuration.
Misc
Thanks
Thanks to the Apache Kafka OpenSource community for their continuous efforts in making the eco-system great. Thanks to the NASA for having a public cluster to run tests with
Note
Inspired by kafka-idle-topics, yet completely re-written to be a continuous monitoring of the topics, similar to cruise-control.
Status
Images build status
Docs build status
1 maintainer
Metadata
1 maintainer
Release files for kafka-overwatch 0.5.0.post0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| kafka_overwatch-0.5.0.post0.tar.gz | 44.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| kafka_overwatch-0.5.0.post0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 114.4 kB
Release files / kafka_overwatch-0.5.0.post0.tar.gz
| Download URL | kafka_overwatch-0.5.0.post0.tar.gz |
|---|---|
| Size | 44.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
05be02ab6616a43c345ced8796f6ef12580b4a93d23ef8df490393a753ed7e89
|
|
BLAKE2b-256 checksum How to use checksums |
c405f211b2bf0745b9ec9f48a96ba23c7e17021dcc570c5527882a2661377695
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.8.2 CPython/3.12.2 Linux/6.5.6-300.fc39.x86_64
|
Release files / kafka_overwatch-0.5.0.post0-py3-none-any.whl
| Download URL | kafka_overwatch-0.5.0.post0-py3-none-any.whl |
|---|---|
| Size | 69.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9df30ff754b541b1c700b34ea4a14cfecfdcfb2b4886bca7f9b78a3ee191268f
|
|
BLAKE2b-256 checksum How to use checksums |
e712ed40e1661d8e61e3256a45321c2d00c86c5ae327328c3ba8776b522c6503
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.8.2 CPython/3.12.2 Linux/6.5.6-300.fc39.x86_64
|
