esque - an operational kafka tool.
esque - an operational Kafka tool
In the Kafka world nothing is easy, but
esque (pronounced esk) is an attempt at it.
esque is a user-centric command line interface for Kafka administration.
Why should you care?
Some stuff is hard, and that is okay, but listing your kafka topics shouldn't be.
While adopting kafka at real.digital we noticed the immense entry barrier it poses to newcomers.
We can't recount how often we wrote Slack messages asking for the script to check the
status of topics or consumer groups. This is partly (but not only) due to a
fragmented and unclear definition of tooling and APIs for kafka.
In a wide array of administration tools,
esque distances itself by striving to provide Kafka Ops for Humans, in a usable and natural way.
We feel that the goal of
esque embodies the principle: “keep easy things easy, and make hard things possible”.
- batteries included
- feature rich
- by engineers for engineers
- Support for any type of Kafka deployment >1.2
- Display Resources (Topics, Consumer Groups, Brokers)
- Get detailed Overviews of Resources (Topics, Consumer Groups, Brokers)
- Create/Delete Topics
- Edit Topic Configurations
- Edit Consumer Offset for Topics
- SASL/SSL Support out of the box
- Consume and Produce to and from Avro and Plaintext Topics (including Avro Schema Resolution from Schema Registry)
- Context Switch (Easily Switch between pre-defined Clusters)
- Kafka Ping (Test roundtrip time to your kafka cluster)
$ esque Usage: esque [OPTIONS] COMMAND [ARGS]... esque - an operational kafka tool. In the Kafka world nothing is easy, but esque (pronounced esk) is an attempt at it. Options: --recreate-config Overwrites the config with the sample config. --version Show the version and exit. -v, --verbose Return stack trace on error. --no-verify Skip all verification dialogs and answer them with yes. --help Show this message and exit. Commands: apply Apply a set of topic configurations. config Configuration-related options. consume Consume messages from a topic. create Create a new instance of a resource. ctx List contexts and switch between them. delete Delete a resource. describe Get detailed information about a resource. edit Edit a resource. get Get a quick overview of different resources. ping Test the connection to the kafka cluster. produce Produce messages to a topic. set Set resource attributes.
Installation and Usage
esque is available at pypi.org and can be installed with
pip install esque.
esque requires Python 3.6+ to run.
When your cluster is secured with SASL authentication, you'll need to install our fork of pykafka since pykafka itself doesn't support it. We've opened a pull request https://github.com/Parsely/pykafka/pull/972 but at the time of writing it hasn't been merged yet.
pip install -U git+https://github.com/real-digital/pykafka.git@feature/sasl-scram-support
esque will also prompt you with the above command as soon as you need it in case you're not sure if you actually do.
The autocompletion scripts for
zsh can be generated by running
esque config autocomplete.
esque for the first time the following message will appear:
No config provided in ~/.esque Should a sample file be created in ~/.esque [y/N]:
When answering with
esque will copy over the sample config to
Afterwards you can modify that file to fit your cluster definitions.
Alternatively might just provide a config file following the sample config's file in that path.
version: 1 current_context: local contexts: # This context corresponds to a local development cluster # created by docker-compose when running esque from the host machine. local: bootstrap_servers: - localhost:9092 security_protocol: PLAINTEXT schema_registry: http://localhost:8081 default_values: num_partitions: 1 replication_factor: 1
Config file for "apply" command
The config for the apply command has to be a yaml file and is given with the option -f or --file.
In the current version only topic configurations can be changed and specified.
It has to use the same schema, which is used for the following example:
topics: - name: topic_one replication_factor: 3 num_partitions: 50 config: cleanup.policy: compact - name: topic_two replication_factor: 3 num_partitions: 50 config: cleanup.policy: compact
To setup your development environment, make sure you have at least Python 3.6 & poetry installed, then run
poetry install poetry run -- pip install -U git+https://github.com/real-digital/pykafka.git@feature/sasl-scram-support poetry shell
Pre Commit Hooks
To install pre commit hooks run:
pip install pre-commit pre-commit install pre-commit install-hooks
esque comes with a docker-compose based kafka stack which you can start up with
You can then run the integration tests against this stack with
pytest tests/ --integration --local.
Alternatively you can go the fast way and just run the whole stack + integration tests in docker:
If you only want the unit tests, just run:
Release history Release notifications
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size esque-0.2.0b2-py3-none-any.whl (63.3 kB)||File type Wheel||Python version py3||Upload date||Hashes View hashes|
|Filename, size esque-0.2.0b2.tar.gz (50.2 kB)||File type Source||Python version None||Upload date||Hashes View hashes|