Skip to main content

Python Stream processing backed by Apache Kafka.

Project description

kaflow

Kafka streams topic processing in Python.

Test


kaflow is a simple framework that allows you to build Kafka streams processing aplications in Python with ease.

Some of the features offered by kaflow:

  • Dependency Injection system inspired by FastAPI and xpresso, and backed by di.
  • Automatic deserialization of incoming messages and serialization of outgoing messages. Supports popular formats like JSON, Avro or Protobuf.
  • Message validation thanks to pydantic.

Requirements

Python 3.8+

Installation

pip install kaflow

Example

from kaflow import (
    FromHeader,
    FromKey,
    FromValue,
    Json,
    Kaflow,
    Message,
    MessageOffset,
    MessagePartition,
    MessageTimestamp,
    String,
)
from pydantic import BaseModel


class UserClick(BaseModel):
    user_id: int
    url: str
    timestamp: int


class Key(BaseModel):
    environment: str


app = Kaflow(name="AwesomeKakfaApp", brokers="localhost:9092")


@app.consume(topic="user_clicks", sink_topics=["user_clicks_json"])
async def consume_user_clicks(
    message: FromValue[Json[UserClick]],
    key: FromKey[Json[Key]],
    x_correlation_id: FromHeader[String[str]],
    x_request_id: FromHeader[String[str]],
    partition: MessagePartition,
    offset: MessageOffset,
    timestamp: MessageTimestamp,
) -> Message:
    # Do something with the message
    ...

    # Publish to another topic
    return Message(value=b'{"user_clicked": "true"}')


app.run()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kaflow-0.2.2.tar.gz (21.1 kB view details)

Uploaded Source

Built Distribution

kaflow-0.2.2-py3-none-any.whl (22.9 kB view details)

Uploaded Python 3

File details

Details for the file kaflow-0.2.2.tar.gz.

File metadata

  • Download URL: kaflow-0.2.2.tar.gz
  • Upload date:
  • Size: 21.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for kaflow-0.2.2.tar.gz
Algorithm Hash digest
SHA256 a8dd0b54b3ef24260077d4a6509463a13cc5cd1ca14db91569dcde242035409c
MD5 8db614c52c018b342e2d8bad745c0758
BLAKE2b-256 8c75da25045f7ff91faa5d52e7d94cdb522bb295bf7ae21d08a4c21557df065e

See more details on using hashes here.

File details

Details for the file kaflow-0.2.2-py3-none-any.whl.

File metadata

  • Download URL: kaflow-0.2.2-py3-none-any.whl
  • Upload date:
  • Size: 22.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for kaflow-0.2.2-py3-none-any.whl
Algorithm Hash digest
SHA256 57a21b8a7e25dbc81d5385f7292204afc05dd6399ff982d7926bfde0b57f3aec
MD5 8056bd1cc6e660039511e513e2cf398e
BLAKE2b-256 56423e50b98b19fda9b34cfe3eca666920e956fc0ef5710dbc5b3fee35043874

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page