KafkaLogHandler
Provides a python logging compatible handler for producing messages to a Kafka message bus.
Depends on the confluent_kafka module to connect to Kafka.
Designed to support both standard and structlog formats, and serializes log data as JSON when published as a Kafka message. Messages are normalized to be more compatible with Logstash/Filebeat formats.
Usage
Example:
import logger
from kafkaloghandler import KafkaLogHandler
log = logging.getLogger()
klh = KafkaLogHandler(bootstrap_servers=["test-kafka:9092"], topic="testtopic")
log.addHandler(klh)
data={'example':'structured data'}
log.info('message to send to kafka', data=data)
Parameters that can be provided to KafkaLogHandler:
- bootstrap_servers
List of Kafka bootstrap servers to connect to.
default: ["localhost:9092"]
- extra_config
Dictionary of extra producer configuration passed to librdkafka.
NOTE: The bootstrap_servers parameter will overwrite bootstrap.servers.
default: {}
- timeout
Timeout in seconds for flushing producer queue. See librdkafka docs.
default: 10.0
- topic
String that sets the topic in Kafka.
default: "kafkaloghandler"
- key
String that sets the default key in Kafka, can be used for summarization within Kafka.
NOTE: This default key can be overridden on a per-message basis by passing a dict to the logger with {"key": "new_key_for_this_message"} in it.
default: "klh"
- flatten
Flattens nested dictionaries and lists passed as structured logging into the parent dictionary layer, up to a certain depth.
This is useful when logging to external systems that don’t have good support for hierarchical data.
Example dictionary: {'a': {'b': 'c'}} would be flattened to {'a.b': 'c'}
Example list: {'a': ['b', 'c']} would be flattened to {'a.0': 'b', 'a.1': 'c'}
If the depth is exceeded, any remaining deeper items will be added to the output under the flattened key.
Set to 0 to turn off flattening.
default: 5
- separator
Separator used between items when flattening.
default: .
- blacklist
List of top-level keys to discard from structured logs when outputting JSON.
default: ["_logger", "_name"]
Testing
Unit tests can be run with tox
Metadata
Release files for kafkaloghandler 0.9.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| kafkaloghandler-0.9.0.tar.gz | 4.8 kB | Without provenance | Details for kafkaloghandler-0.9.0.tar.gz |
Release files / kafkaloghandler-0.9.0.tar.gz
| Download URL | kafkaloghandler-0.9.0.tar.gz |
|---|---|
| Size | 4.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cdca2ba180616f806ee713184587efdfa105d4cd96b581fc424e1228a3eb7f42
|
|
BLAKE2b-256 checksum How to use checksums |
8f1d6cbc7618da6dd2a36db5dd665a57283520cea41d2bec750f058eebf3c689
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.11.0 pkginfo/1.4.2 requests/2.20.0 setuptools/40.4.2 requests-toolbelt/0.8.0 tqdm/4.27.0 CPython/2.7.15
|
Security for kafkaloghandler 0.9.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.