Kafka Logging Handler
Project description
KafkaLogHandler
Provides a python logging compatible handler for producing messages to a Kafka message bus.
Depends on the confluent_kafka module to connect to Kafka.
Designed to support both standard and structlog formats, and serializes log data as JSON when published as a Kafka message. Messages are normalized to be more compatible with Logstash/Filebeat formats.
Usage
Example:
import logger from kafkaloghandler import KafkaLogHandler log = logging.getLogger() klh = KafkaLogHandler(bootstrap_servers=["test-kafka:9092"], topic="testtopic") log.addHandler(klh) data={'example':'structured data'} log.info('message to send to kafka', data=data)
Parameters that can be provided to KafkaLogHandler:
- bootstrap_servers
List of Kafka bootstrap servers to connect to. See confluent_kafka docs.
default: ["localhost:9092"]
- timeout
Timeout in seconds for flushing producer queue. See confluent_kafka docs.
default: 10.0
- topic
String that sets the topic in Kafka.
default: "kafkaloghandler"
- key
String that sets the default key in Kafka, can be used for summarization within Kafka.
NOTE: This default key can be overridden on a per-message basis by passing a dict to the logger with {"key": "new_key_for_this_message"} in it.
default: "klh"
- flatten
Flattens nested dictionary keys passed as structured logging into the parent dictionary layer, up to a certain depth.
This is useful when logging to external systems that don’t have good support for hierarchical data.
Example: {'a': {'b': 'c'}} would be flattened to {'a.b': 'c'}
If the depth is exceeded, any remaining deeper dict will be added to the output under the flattened key.
Set to 0 to turn off flattening.
default: 5
- separator
Separator used between keys when flattening.
default: .
- blacklist
List of top-level keys to discard from structured logs when outputting JSON.
default: ["_logger", "_name"]
Tests
Unit tests can be run with:
nose2 –verbose –coverage-report term
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.