Streamline your Kafka data processing, this tool aims to standardize streaming data from multiple Kafka clusters. With a pub-sub approach, multiple functions can easily subscribe to incoming messages, serialization can be specified per topic, and data is automatically processed by data sink functions.
Project description
Snapstream
Snapstream provides a data-flow model to simplify development of stateful streaming applications.
Installation
pip install snapstream
Usage
We snap
iterables to user functions, and process them in parallel when we call stream
:
We pass the callable print
to print out the return value. Multiple iterables and sinks can be passed.
from snapstream import snap, stream
@snap(range(5), sink=[print])
def handler(msg):
yield f'Hello {msg}'
stream()
Hello 0
Hello 1
Hello 2
Hello 3
Hello 4
To try it out for yourself, spin up a local kafka broker with docker-compose.yml, using localhost:29091
to connect:
docker compose up broker -d
Use the cli tool to inspect Topic/Cache:
snapstream topic emoji --offset -2
>>> timestamp: 2023-04-28T17:31:51.775000+00:00
>>> offset: 0
>>> key:
🏆
Features
snapstream.snap
: bind streams (iterables) and sinks (callables) to user defined handler functionssnapstream.stream
: start streamingsnapstream.Topic
: consume from (iterable), and produce to (callable) kafka using confluent-kafkasnapstream.Cache
: store data to disk using rocksdictsnapstream.Conf
: set global kafka configuration (can be overridden per topic)snapstream.codecs.AvroCodec
: serialize and deserialize avro messagessnapstream.codecs.JsonCodec
: serialize and deserialize json messages
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for snapstream-1.0.2b0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29392d5c0250a960e2c847d08a02d4dc69fa6bec2e873a16f9bd52e59459bd53 |
|
MD5 | ed252154f9678b96d31b3c877c6fe707 |
|
BLAKE2b-256 | ca0d2164a4009af8987b33d37ad444c635ff5634bf7bbe424f121c8010528390 |