Skip to main content

Kstreams

kstreams is a library/micro framework to use with kafka. It has simple kafka streams implementation that gives certain guarantees, see below.

Build status codecov python version


Documentation: https://kpn.github.io/kstreams/


Installation

pip install kstreams

You will need a worker, we recommend aiorun

pip install aiorun

Usage

import aiorun
from kstreams import create_engine, ConsumerRecord


stream_engine = create_engine(title="my-stream-engine")

@stream_engine.stream("local--kstream")
async def consume(cr: ConsumerRecord):
    print(f"Event consumed: headers: {cr.headers}, payload: {cr.value}")


async def produce():
    payload = b'{"message": "Hello world!"}'

    for i in range(5):
        metadata = await stream_engine.send("local--kstreams", value=payload)
        print(f"Message sent: {metadata}")


async def start():
    await stream_engine.start()
    await produce()


async def shutdown(loop):
    await stream_engine.stop()


if __name__ == "__main__":
    aiorun.run(start(), stop_on_unhandled_errors=True, shutdown_callback=shutdown)

Features

  • Produce events
  • Consumer events with Streams
  • Subscribe to topics by pattern
  • Prometheus metrics and custom monitoring
  • TestClient
  • Custom Serialization and Deserialization
  • Easy to integrate with any async framework. No tied to any library!!
  • Yield events from streams
  • Opentelemetry Instrumentation
  • Middlewares
  • Hooks (on_startup, on_stop, after_startup, after_stop)
  • Store (kafka streams pattern)
  • Stream Join
  • Windowing

Development

This repo requires the use of poetry instead of pip. Note: If you want to have the virtualenv in the same path as the project first you should run poetry config --local virtualenvs.in-project true

To install the dependencies just execute:

poetry install

Then you can activate the virtualenv with

poetry shell

Run test:

./scripts/test

Run code formatting with ruff:

./scripts/format

Commit messages

We use conventional commits for the commit message.

The use of commitizen is recommended. Commitizen is part of the dev dependencies.

cz commit

Metadata

Release files for kstreams 0.34.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kstreams 0.34.6
File Size Uploaded Provenance Actions
kstreams-0.34.6.tar.gz 36.0 kB Without provenance Details for kstreams-0.34.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for kstreams 0.34.6
File Interpreter ABI Platform Actions
kstreams-0.34.6-py3-none-any.whl Python 3 none any Details for kstreams-0.34.6-py3-none-any.whl

Total release size: 81.6 kB

Release files / kstreams-0.34.6.tar.gz

Download URL kstreams-0.34.6.tar.gz
Size 36.0 kB
Tags Source
SHA-256 checksum
How to use checksums
ca547b03ce4b6a48e05e7a2ebd766e8cd97ab9cd57adf2c942a63f79671827df
BLAKE2b-256 checksum
How to use checksums
99cab05912bb166ebfca7eb8fda37ef3c8ecc18ea2ec9cf59ec7368622420080
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.14.8 Linux/6.17.0-1022-azure

Release files / kstreams-0.34.6-py3-none-any.whl

Download URL kstreams-0.34.6-py3-none-any.whl
Size 45.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
80ef84d231f45bddd68b075abc1da8eaf69c1d69b77958da7a683725312fc5ee
BLAKE2b-256 checksum
How to use checksums
5fe04a7d8e3592c3e1d44ef633e14c781bb53ffe0aafce65eaec30ffbd9a6cce
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.14.8 Linux/6.17.0-1022-azure

Security for kstreams 0.34.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.34.6 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page