kser 0.8.27

Kafka serialize python library

Kafka serialize python library

Kser is a bundle of python library whose purpose is to serialize tasks to be executed on Kafka consumers. It supports farious extensions:

Transport

librdkafka

You can choose to use the C bindings using confluent-kafka:

$ pip install kser[confluent]

Note

You need to install manually librdkafka, see confluent-kafka-python documentation

kafka-python

You can choose the pure python library kafka-python:

$ pip install kser[pykafka]

http

There is also a light version using HTTP ideal to produce messages (not recommended for consumption)

$ pip install kser[http]

dummy

There is a dummy version for tests, no additional import required, just use / patch with the dummy module.

Other

encrypt data

It is possible to encrypt messages in the Kafka queue using libsodium.

$ pip install kser[crypto]

Note

You need to install manually libsodium, see libsodium documentation

prometheus exporter

You can export metrics using the prometheus format:

$ pip install kser[prometheus]

A few set of environment variables allow to manage the exporter:

Environment variable	Default value
KSER_METRICS_ENABLED	no
KSER_METRICS_ADDRESS	0.0.0.0
KSER_METRICS_PORT	8888

The exporter has only 2 metrics defined by default, it’s just a sample. A good way to implement your own is to override the triggers methods (prefixed with ‘_’) like the following example:

from kser import KSER_METRICS_ENABLED
from prometheus_client import Counter
from kser.entry import Entrypoint

MY_METRIC = Counter('kser_my_metric', 'a usefull metric')


class MyEntrypoint(Entrypoint):
    def _run(self):
        if KSER_METRICS_ENABLED == "yes":
            MY_METRIC.inc()

        return self.run()

See also:

prometheus_client documentation

Documentation of the python Prometheus client

opentracing support

You can use opentracing with Jaeger to follow operation and task:

$ pip install kser[opentracing]

Configuration can be set using environment variable:

Environment variable	Default value
JAEGER_HOST	localhost

See also:

Opentracing support with Jaeger

Kser documentation

Add-ons

• Flask extension: Flask integration.

Requirements

• Python 3.x

Documentations

• Project: http://kser.readthedocs.io/

• Libsodium: https://download.libsodium.org/doc/

• confluent-kafka-python: http://docs.confluent.io/current/clients/confluent-kafka-python

• kafka-python: http://kafka-python.readthedocs.io/en/master/

Other links

• PyPI: https://pypi.python.org/pypi/kser

• Project issues: https://github.com/cdumay/kser/issues

License

Licensed under BSD 3-Clause License or https://opensource.org/licenses/BSD-3-Clause.