Build simple kafka streams applications
Project description
Kstreams
kstreams
is a library/micro framework to use with kafka
. It has simple kafka streams implementation that gives certain guarantees, see below.
Requirements
python 3.8+
API documentation
https://kpn.github.io/kstreams/
Installation
pip install kstreams
Usage
import asyncio
from kstreams import create_engine, Stream
stream_engine = create_engine(title="my-stream-engine")
@stream_engine.stream("local--kstream")
async def consume(stream: Stream):
async for cr in stream:
print(f"Event consumed: headers: {cr.headers}, payload: {cr.value}")
async def produce():
payload = b'{"message": "Hello world!"}'
for i in range(5):
metadata = await create_engine.send("local--kstreams", value=payload)
print(f"Message sent: {metadata}")
await asyncio.sleep(3)
async def main():
await stream_engine.start()
await produce()
await stream_engine.stop()
if __name__ == "__main__":
asyncio.run(main())
Features
- Produce events
- Consumer events with
Streams
-
Prometheus
metrics and custom monitoring - TestClient
- Custom Serialization and Deserialization
- Easy to integrate with any
async
framework. No tied to any library!! - Yield events from streams
- Store (kafka streams pattern)
- Stream Join
- Windowing
Development
This repo requires the use of poetry instead of pip.
Note: If you want to have the virtualenv
in the same path as the project first you should run poetry config --local virtualenvs.in-project true
To install the dependencies just execute:
poetry install
Then you can activate the virtualenv
with
poetry shell
Run test:
./scripts/test
Run code linting (black
and isort
)
./scripts/lint
Commit messages
The use of commitizen is recommended. Commitizen is part of the dev dependencies.
cz commit
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
kstreams-0.2.2.tar.gz
(16.6 kB
view hashes)
Built Distribution
kstreams-0.2.2-py3-none-any.whl
(20.2 kB
view hashes)