Automating the creation of topics and ACLs is a common task for Kafka administrators. This repository contains a set of utilities to automate these tasks.
Project description
Kafka Ease
Automating the creation of topics and ACLs is a common task for Kafka administrators. This repository contains a set of utilities to automate these tasks.
- Create a topic with a given number of partitions and replication factor
- Create ACLs for a given topic
See the documentation for more information.
Contents
Getting started
Installation
requirements.txt
To install the library from the GitLab package registry, write the following lines in the file requirements.txt
:
kafka-ease==[VERSION]
Pip install
To install the library, run the following command:
pip install kafka-ease==[VERSION]
Check PEP 440 for version schemes and version specifiers.
How to use
File format
The file format is YAML or JSON. The file must contain a list of topics and a list of ACLs.
# acl.yaml
topics:
- name: topic.name
num_partitions: 1
replication_factor: 1
cleanup_policy: delete
retention_days: 7
acls:
- resource_type: TOPIC
resource_name: topic.name
principal: User:kafka-user
host: "*"
operation: READ
permission_type: ALLOW
pattern_type: LITERAL
{
"topics": [
{
"name": "topic.name",
"num_partitions": 1,
"replication_factor": 1,
"cleanup_policy": "delete",
"retention_days": 7
}
],
"acls": [
{
"resource_type": "TOPIC",
"resource_name": "topic.name",
"principal": "User:kafka-user",
"host": "*",
"operation": "READ",
"permission_type": "ALLOW",
"pattern_type": "LITERAL"
}
]
}
-
Topics
- name: Topic name
- num_partitions: Number of partitions
- replication_factor: Replication factor
- cleanup_policy: Cleanup policy (
delete
,compact
) - retention_days: Delete retention days
-
ACLs
- resource_type: Resource type (
UNKNOWN
,ANY
,CLUSTER
,TOPIC
,DELEGATION_TOKEN
,GROUP
,TRANSACTIONAL_ID
) - resource_name: Resource name
- principal: Principal (
User:
,Group:
) - host: Host (
*
) - operation: Operation (
ANY
,ALL
,READ
,WRITE
,CREATE
,DELETE
,ALTER
,DESCRIBE
,CLUSTER_ACTION
,DESCRIBE_CONFIGS
,ALTER_CONFIGS
,IDEMPOTENT_WRITE
) - permission_type: Permission type (
ANY
,DENY
,ALLOW
) - pattern_type: Pattern type (
ANY
,MATCH
,LITERAL
,PREFIXED
)
- resource_type: Resource type (
Validate configuration
kafka-ease apply -f acl.yaml --only-validate
Expected output:
Validating file acl.yaml...
File: acl.yaml
YAML file detected
2 topics found.
2 ACLs found.
File acl.yaml is valid.
Apply configuration
kafka-ease apply -f acl.yaml \
--kafka-brokers kafka:9093 \
--security-protocol SASL_PLAINTEXT \
--sasl-mechanism SCRAM-SHA-256 \
--sasl-username kafka-admin \
--sasl-password SECRET
Expected output:
Applying file...
Kafka brokers: kafka:9093
Security protocol: SASL_PLAINTEXT
SASL mechanism: SCRAM-SHA-256
SASL username: kafka-admin
SASL password: ***************
File: acl.yaml
YAML file detected
1 topics found.
1 ACLs found.
Topic topic.name updated
Removing old ACLs 3
ACL User:kafka-user synced
File synced successfully.
Help
kafka-ease apply --help
Expected output:
Usage: kafka-ease apply [OPTIONS]
Apply configuration file with topics and ACL to Kafka.
Options:
--kafka-brokers TEXT Kafka server to connect to use.
--security-protocol [PLAINTEXT|SASL_PLAINTEXT]
Kafka Security protocol to use.
--sasl-mechanism [PLAIN|SCRAM-SHA-256|SCRAM-SHA-512]
SASL mechanism to use.
--sasl-username TEXT SASL username to use.
--sasl-password TEXT SASL password to use.
-f, --file TEXT File path to validate
--only-validate Only validate the file
--help Show this message and exit.
Changelog
- 1.0.0 (2023-09-20)
Initial release of Kafka Ease
- 1.0.1 (2023-09-20)
Fix packaging error
- 1.0.2 (2023-09-20)
Fix minor bugs
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for kafka_ease-1.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 27dcbd33992795e7bde95365e859db4a78df392342f1cb2bf669ebc1b3a9c8ea |
|
MD5 | 2fbb27db9e8cd2b831f63d0e023a9009 |
|
BLAKE2b-256 | 3afc7c1269d0b2c8c1bf2ced70ac06cae4cc9e68a39520cdb781e43d520036a4 |