Skip to main content

Kstreams

kstreams is a library/micro framework to use with kafka. It has simple kafka streams implementation that gives certain guarantees, see below.

Build status codecov python version


Documentation: https://kpn.github.io/kstreams/


Installation

pip install kstreams

You will need a worker, we recommend aiorun

pip install aiorun

Usage

import aiorun
from kstreams import create_engine, ConsumerRecord


stream_engine = create_engine(title="my-stream-engine")

@stream_engine.stream("local--kstream")
async def consume(cr: ConsumerRecord):
    print(f"Event consumed: headers: {cr.headers}, payload: {cr.value}")


async def produce():
    payload = b'{"message": "Hello world!"}'

    for i in range(5):
        metadata = await stream_engine.send("local--kstreams", value=payload)
        print(f"Message sent: {metadata}")


async def start():
    await stream_engine.start()
    await produce()


async def shutdown(loop):
    await stream_engine.stop()


if __name__ == "__main__":
    aiorun.run(start(), stop_on_unhandled_errors=True, shutdown_callback=shutdown)

Features

  • Produce events
  • Consumer events with Streams
  • Subscribe to topics by pattern
  • Prometheus metrics and custom monitoring
  • TestClient
  • Custom Serialization and Deserialization
  • Easy to integrate with any async framework. No tied to any library!!
  • Yield events from streams
  • Opentelemetry Instrumentation
  • Middlewares
  • Hooks (on_startup, on_stop, after_startup, after_stop)
  • Store (kafka streams pattern)
  • Stream Join
  • Windowing

Development

This repo requires the use of poetry instead of pip. Note: If you want to have the virtualenv in the same path as the project first you should run poetry config --local virtualenvs.in-project true

To install the dependencies just execute:

poetry install

Then you can activate the virtualenv with

poetry shell

Run test:

./scripts/test

Run code formatting with ruff:

./scripts/format

Commit messages

We use conventional commits for the commit message.

The use of commitizen is recommended. Commitizen is part of the dev dependencies.

cz commit

Metadata

Release files for kstreams 0.35.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kstreams 0.35.0
File Size Uploaded Provenance Actions
kstreams-0.35.0.tar.gz 36.4 kB Without provenance Details for kstreams-0.35.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for kstreams 0.35.0
File Interpreter ABI Platform Actions
kstreams-0.35.0-py3-none-any.whl Python 3 none any Details for kstreams-0.35.0-py3-none-any.whl

Total release size: 81.9 kB

Release files / kstreams-0.35.0.tar.gz

Download URL kstreams-0.35.0.tar.gz
Size 36.4 kB
Tags Source
SHA-256 checksum
How to use checksums
4b27f2b6a5e0db7453589744031fbd2e15c552d2518e5df262149519298df2d9
BLAKE2b-256 checksum
How to use checksums
65ef7d6c2236eb43a837e66d575a8a79c8ad163023abae9fe739e0483f47a692
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.15.2 Linux/7.0.0-1019-azure

Release files / kstreams-0.35.0-py3-none-any.whl

Download URL kstreams-0.35.0-py3-none-any.whl
Size 45.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
561d991f08862ffb5516aa7dbfd29648eb4f194253f48b827da4ca94e80377ff
BLAKE2b-256 checksum
How to use checksums
244985e62fed2f24f8ee03aeda8d9e42bca83494fb00132203a678a4b23683aa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.5.1 CPython/3.15.2 Linux/7.0.0-1019-azure

Security for kstreams 0.35.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.35.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page