Skip to main content

Kafka serialize python library

Project description

Latest Version Latest version Documentation Status https://img.shields.io/badge/license-MIT-blue.svg

Kafka serialize python library

Kser is a bundle of python library whose purpose is to serialize tasks to be executed on Kafka consumers. It supports farious extensions:

Transport

librdkafka

You can choose to use the C bindings using confluent-kafka:

$ pip install kser[confluent]

kafka-python

You can choose the pure python library kafka-python:

$ pip install kser[pykafka]

http

There is also a light version using HTTP ideal to produce messages (not recommended for consumption)

$ pip install kser[http]

Other

encrypt data

It is possible to encrypt messages in the Kafka queue using libsodium.

$ pip install kser[crypto]

prometheus exporter

You can export metrics using the prometheus format:

$ pip install kser[prometheus]

A few set of environment variables allow to manage the exporter:

Environment variable

Default value

KSER_METRICS_ENABLED

no

KSER_METRICS_ADDRESS

0.0.0.0

KSER_METRICS_PORT

8888

The exporter has only 2 metrics defined by default, it’s just a sample. A good way to implement your own is to override the triggers methods (prefixed with ‘_’) like the following example:

from kser import KSER_METRICS_ENABLED
from prometheus_client import Counter
from kser.entry import Entrypoint

MY_METRIC = Counter('kser_my_metric', 'a usefull metric')


class MyEntrypoint(Entrypoint):
    def _run(self):
        if KSER_METRICS_ENABLED == "yes":
            MY_METRIC.inc()

        return self.run()

See also:

prometheus_client documentation

Documentation of the python Prometheus client

opentracing support

You can use opentracing with Jaeger to follow operation and task:

$ pip install kser[opentracing]

Configuration can be set using environment variable:

Environment variable

Default value

JAEGER_HOST

localhost

See also:

Opentracing support with Jaeger

Kser documentation

Add-ons

Requirements

  • Python 3.x

Documentations

License

Licensed under MIT license (LICENSE or http://opensource.org/licenses/MIT)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kser-0.8.24.tar.gz (13.8 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page