Skip to main content

A collection of tools and scripts for working with Apache Kafka

Project description

This repository is a collection of tools and scripts for working with Apache Kafka <http://kafka.apache.org>. The Site Reliability team for Kafka at LinkedIn has built these over time in order to make managing Kafka a little bit easier. Our intention is to add to this repository as more tools are developed, and we welcome additions and modifications that make things better for all!

Build Status Code Climate Code Climate Test Coverage Coveralls Test Coverage Codecov Issue Count

Current Commands

  • kafka-assigner - This script is used for performing partition reassignments and preferred replica elections. It uses the admin CLI utilities provided with Kafka and layers on additional logic to perform tasks like removing a broker, rebalancing partitions, fixing partition replcation factors, and performing preferred replica elections.

Prerequisites

These tools are generally written in Python, and besides a basic installation, you will need the following additional modules:

  • Paramiko
  • Kazoo

In addition, you will need to run it on a host that has the following:

  • A copy of the Kafka admin tools (including kafka-reassign-partitions.sh).
  • Access to the Zookeeper ensemble for the cluster.
  • SSH access to the Kafka brokers (with credentials preferably loaded into ssh-agent).

Contributing

We’re always open to fixes and new features! Please open a PR for any changes that you have and someone will review and merge it. If you’re not up for writing the code, open an issue for any problems or requests.

Other Projects

In addition to these tools, LinkedIn has also open-sourced Burrow <https://github.com/linkedin/Burrow>, a robust system for monitoring Kafka consumer clients.

License

Copyright 2016 LinkedIn Corp. Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
kafka-tools-0.0.1.zip (64.6 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page