Skip to main content

Automating the creation of topics and ACLs is a common task for Kafka administrators. This repository contains a set of utilities to automate these tasks.

Project description

Kafka Ease

Automating the creation of topics and ACLs is a common task for Kafka administrators. This repository contains a set of utilities to automate these tasks.

  • Create a topic with a given number of partitions and replication factor
  • Create ACLs for a given topic

coverage badge

CI

See the documentation for more information.

Contents

Getting started

Installation

requirements.txt

To install the library from the GitLab package registry, write the following lines in the file requirements.txt:

kafka-ease==[VERSION]

Pip install

To install the library, run the following command:

pip install kafka-ease==[VERSION]

Check PEP 440 for version schemes and version specifiers.

How to use

File format

The file format is YAML or JSON. The file must contain a list of topics and a list of ACLs.

# acl.yaml
topics:
  - name: topic.name
    num_partitions: 1
    replication_factor: 1
    cleanup_policy: delete
    retention_days: 7

acls:
  - resource_type: TOPIC
    resource_name: topic.name
    principal: User:kafka-user
    host: "*"
    operation: READ
    permission_type: ALLOW
    pattern_type: LITERAL
{
  "topics": [
    {
      "name": "topic.name",
      "num_partitions": 1,
      "replication_factor": 1,
      "cleanup_policy": "delete",
      "retention_days": 7
    }
  ],
  "acls": [
    {
      "resource_type": "TOPIC",
      "resource_name": "topic.name",
      "principal": "User:kafka-user",
      "host": "*",
      "operation": "READ",
      "permission_type": "ALLOW",
      "pattern_type": "LITERAL"
    }
  ]
}
  • Topics

    • name: Topic name
    • num_partitions: Number of partitions
    • replication_factor: Replication factor
    • cleanup_policy: Cleanup policy (delete, compact)
    • retention_days: Delete retention days
  • ACLs

    • resource_type: Resource type (UNKNOWN, ANY, CLUSTER, TOPIC, DELEGATION_TOKEN, GROUP, TRANSACTIONAL_ID)
    • resource_name: Resource name
    • principal: Principal (User:, Group:)
    • host: Host (*)
    • operation: Operation (ANY, ALL, READ, WRITE, CREATE, DELETE, ALTER, DESCRIBE, CLUSTER_ACTION, DESCRIBE_CONFIGS, ALTER_CONFIGS, IDEMPOTENT_WRITE)
    • permission_type: Permission type (ANY, DENY, ALLOW)
    • pattern_type: Pattern type (ANY, MATCH, LITERAL, PREFIXED)

Validate configuration

kafka-ease apply -f acl.yaml --only-validate

Expected output:

Validating file acl.yaml...
File: acl.yaml
YAML file detected
2 topics found.
2 ACLs found.
File acl.yaml is valid.

Apply configuration

kafka-ease apply -f acl.yaml \
  --kafka-brokers kafka:9093 \
  --security-protocol SASL_PLAINTEXT \ 
  --sasl-mechanism SCRAM-SHA-256 \
  --sasl-username kafka-admin \
  --sasl-password SECRET

Expected output:

Applying file...
Kafka brokers: kafka:9093
Security protocol: SASL_PLAINTEXT
SASL mechanism: SCRAM-SHA-256
SASL username: kafka-admin
SASL password: ***************
File: acl.yaml
YAML file detected
1 topics found.
1 ACLs found.
Topic topic.name updated
Removing old ACLs 3
ACL User:kafka-user synced
File synced successfully.

Help

kafka-ease apply --help

Expected output:

Usage: kafka-ease apply [OPTIONS]

  Apply configuration file with topics and ACL to Kafka.

Options:
  --kafka-brokers TEXT            Kafka server to connect to use.
  --security-protocol [PLAINTEXT|SASL_PLAINTEXT]
                                  Kafka Security protocol to use.
  --sasl-mechanism [PLAIN|SCRAM-SHA-256|SCRAM-SHA-512]
                                  SASL mechanism to use.
  --sasl-username TEXT            SASL username to use.
  --sasl-password TEXT            SASL password to use.
  -f, --file TEXT                 File path to validate
  --only-validate                 Only validate the file
  --help                          Show this message and exit.

Changelog

  • 1.0.0 (2023-09-20)

    Initial release of Kafka Ease

  • 1.0.1 (2023-09-20)

    Fix packaging error

  • 1.0.2 (2023-09-20)

    Fix minor bugs

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kafka-ease-1.0.3.tar.gz (30.8 kB view hashes)

Uploaded Source

Built Distribution

kafka_ease-1.0.3-py3-none-any.whl (21.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page