Skip to main content
Donate to the Python Software Foundation or Purchase a PyCharm License to Benefit the PSF! Donate Now

Kafka serialize python library

Project description

Latest Version Latest version Documentation Status https://img.shields.io/badge/license-MIT-blue.svg

Kafka serialize python library

Kser is a bundle of python library whose purpose is to serialize tasks to be executed on Kafka consumers. It supports farious extensions:

Transport

librdkafka

You can choose to use the C bindings using confluent-kafka:

$ pip install kser[confluent]

Note

You need to install manually librdkafka, see confluent-kafka-python documentation

kafka-python

You can choose the pure python library kafka-python:

$ pip install kser[pykafka]

http

There is also a light version using HTTP ideal to produce messages (not recommended for consumption)

$ pip install kser[http]

Other

encrypt data

It is possible to encrypt messages in the Kafka queue using libsodium.

$ pip install kser[crypto]

Note

You need to install manually libsodium, see libsodium documentation

prometheus exporter

You can export metrics using the prometheus format:

$ pip install kser[prometheus]

A few set of environment variables allow to manage the exporter:

Environment variable Default value
KSER_METRICS_ENABLED no
KSER_METRICS_ADDRESS 0.0.0.0
KSER_METRICS_PORT 8888

The exporter has only 2 metrics defined by default, it’s just a sample. A good way to implement your own is to override the triggers methods (prefixed with ‘_’) like the following example:

from kser import KSER_METRICS_ENABLED
from prometheus_client import Counter
from kser.entry import Entrypoint

MY_METRIC = Counter('kser_my_metric', 'a usefull metric')


class MyEntrypoint(Entrypoint):
    def _run(self):
        if KSER_METRICS_ENABLED == "yes":
            MY_METRIC.inc()

        return self.run()

See also:

prometheus_client documentation
Documentation of the python Prometheus client

opentracing support

You can use opentracing with Jaeger to follow operation and task:

$ pip install kser[opentracing]

Configuration can be set using environment variable:

Environment variable Default value
JAEGER_HOST localhost

See also:

Opentracing support with Jaeger
Kser documentation

Add-ons

Requirements

  • Python 3.x

License

Licensed under MIT license (LICENSE or http://opensource.org/licenses/MIT)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
kser-0.8.19.tar.gz (13.7 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page