kafka-collector

collect kafka messages from multiple topics

Installation

pip install kafka_collector

Usage

$ kafka-collector -h
usage: kafka-collector [-h] [-v] [-t TOPICS] [-b BOOTSTRAP_SERVER] [-g GROUP]
                       [-o OUTPUT] [-c CAPTURE_DIR] [-m {cli,service}]
                       [-p PORT]
                       [--log-level {debug,info,warning,error,critical}]
                       [--debug]

collect kafka messages from multiple topics

options:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -t, --topics TOPICS   comma separated list of kafka topics to be listened to
  -b, --bootstrap-server BOOTSTRAP_SERVER
                        kafka bootstrap server (default: localhost:9092)
  -g, --group GROUP     consumer group id (default: random uuid)
  -o, --output OUTPUT   output file path, use '-' for stdout (default: stdout)
  -c, --capture-dir CAPTURE_DIR
                        capture directory for service mode (default:
                        /tmp/kafka-collector)
  -m, --mode {cli,service}
                        run mode: cli or service (default: cli)
  -p, --port PORT       service port for service mode (default: 8080)
  --log-level {debug,info,warning,error,critical}
                        logging level (default: info)
  --debug               enable debug logging (shortcut for --log-level debug)

Environment Variables

• KAFKA_TOPICS - comma separated list of topics
• KAFKA_BOOTSTRAP_SERVER - bootstrap server address
• KAFKA_GROUP - consumer group id
• COLLECTOR_CAPTURE_DIR - capture directory for service mode
• COLLECTOR_MODE - run mode (cli or service)
• COLLECTOR_SERVICE_PORT - service port (must be integer)
• LOG_LEVEL - logging level

Documentation

• CLI Mode - Default mode for streaming messages to stdout or file
• Service Mode - HTTP service mode with REST API for capture management

Deployment Examples

• Docker Compose - Run with Docker Compose
• Kubernetes - Deploy to Kubernetes

Links

• https://pypi.org/project/kafka_collector/
• https://github.com/siakhooi/kafka-collector
• https://sonarcloud.io/project/overview?id=siakhooi_kafka-collector
• https://qlty.sh/gh/siakhooi/projects/kafka-collector

Badges