Kafka serialize python library
Project description
Kafka serialize python library
Kser is a bundle of python library whose purpose is to serialize tasks to be executed on Kafka consumers. It supports farious extensions:
Transport
librdkafka
You can choose to use the C bindings using confluent-kafka:
$ pip install kser[confluent]
kafka-python
You can choose the pure python library kafka-python:
$ pip install kser[pykafka]
http
There is also a light version using HTTP ideal to produce messages (not recommended for consumption)
$ pip install kser[http]
Other
encrypt data
It is possible to encrypt messages in the Kafka queue using libsodium.
$ pip install kser[crypto]
prometheus exporter
You can export metrics using the prometheus format:
$ pip install kser[prometheus]
A few set of environment variables allow to manage the exporter:
Environment variable |
Default value |
---|---|
KSER_METRICS_ENABLED |
no |
KSER_METRICS_ADDRESS |
0.0.0.0 |
KSER_METRICS_PORT |
8888 |
The exporter has only 2 metrics defined by default, it’s just a sample. A good way to implement your own is to override the triggers methods (prefixed with ‘_’) like the following example:
from kser import KSER_METRICS_ENABLED
from prometheus_client import Counter
from kser.entry import Entrypoint
MY_METRIC = Counter('kser_my_metric', 'a usefull metric')
class MyEntrypoint(Entrypoint):
def _run(self):
if KSER_METRICS_ENABLED == "yes":
MY_METRIC.inc()
return self.run()
Add-ons
Flask extension: Flask integration.
Requirements
Python 3.x
Documentations
Project: http://kser.readthedocs.io/
Libsodium: https://download.libsodium.org/doc/
confluent-kafka-python: http://docs.confluent.io/current/clients/confluent-kafka-python
kafka-python: http://kafka-python.readthedocs.io/en/master/
Other links
Project issues: https://github.com/cdumay/kser/issues
License
MIT license
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.