Python Stream processing backed by Apache Kafka.
Project description
kaflow
is a simple framework that allows you to build Kafka streams processing aplications in Python with ease.
Some of the features offered by kaflow
:
- Dependency Injection system inspired by FastAPI and xpresso, and backed by di.
- Automatic deserialization of incoming messages and serialization of outgoing messages. Supports popular formats like
JSON
,Avro
orProtobuf
. - Message validation thanks to pydantic.
Requirements
Python 3.8+
Installation
pip install kaflow
Example
from kaflow import (
FromHeader,
FromKey,
FromValue,
Json,
Kaflow,
Message,
MessageOffset,
MessagePartition,
MessageTimestamp,
String,
)
from pydantic import BaseModel
class UserClick(BaseModel):
user_id: int
url: str
timestamp: int
class Key(BaseModel):
environment: str
app = Kaflow(name="AwesomeKakfaApp", brokers="localhost:9092")
@app.consume(topic="user_clicks", sink_topics=["user_clicks_json"])
async def consume_user_clicks(
message: FromValue[Json[UserClick]],
key: FromKey[Json[Key]],
x_correlation_id: FromHeader[String[str]],
x_request_id: FromHeader[String[str]],
partition: MessagePartition,
offset: MessageOffset,
timestamp: MessageTimestamp,
) -> Message:
# Do something with the message
...
# Publish to another topic
return Message(value=b'{"user_clicked": "true"}')
app.run()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
kaflow-0.2.2.tar.gz
(21.1 kB
view details)
Built Distribution
kaflow-0.2.2-py3-none-any.whl
(22.9 kB
view details)
File details
Details for the file kaflow-0.2.2.tar.gz
.
File metadata
- Download URL: kaflow-0.2.2.tar.gz
- Upload date:
- Size: 21.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8dd0b54b3ef24260077d4a6509463a13cc5cd1ca14db91569dcde242035409c |
|
MD5 | 8db614c52c018b342e2d8bad745c0758 |
|
BLAKE2b-256 | 8c75da25045f7ff91faa5d52e7d94cdb522bb295bf7ae21d08a4c21557df065e |
File details
Details for the file kaflow-0.2.2-py3-none-any.whl
.
File metadata
- Download URL: kaflow-0.2.2-py3-none-any.whl
- Upload date:
- Size: 22.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 57a21b8a7e25dbc81d5385f7292204afc05dd6399ff982d7926bfde0b57f3aec |
|
MD5 | 8056bd1cc6e660039511e513e2cf398e |
|
BLAKE2b-256 | 56423e50b98b19fda9b34cfe3eca666920e956fc0ef5710dbc5b3fee35043874 |