A mini-framework for Kafka apps
Project description
Kaf
Så er det tid til en kop kaf'
Kaf is a small Python framework for creating Kafka apps. It is inspired by Faust, but differs in the following ways:
- Kaf is synchronous (async is future work)
- Kaf is compatible with Azure Eventhubs (over Kafka interface)
- Kaf is designed to work with different brokers for the consumer and producer
- Kaf expects JSON events
The framework depends on Confluent Kafka.
How to use
Hello world:
import logging
from kaf import KafkaApp
consumer_conf = {'bootstrap.servers': 'kafka:9092', 'group.id': 'myapp'}
producer_conf = {'bootstrap.servers': 'kafka:9092'}
app = KafkaApp('myapp', consumer_conf, producer_conf)
counter_ok = 0
counter_failed = 0
app.logger.setLevel(logging.INFO)
@app.process(topic='foo', publish_to='bar')
def uppercase_everything(msg):
return {key:str(val).upper() for key, val in msg.items()}
@app.on_processed
def inc_ok(msg, seconds_elapsed):
counter_ok += 1
@app.on_failed
def inc_failed(msg, error):
counter_failed += 1
if __name__ == '__main__':
app.run()
How errors are handled
In case of an unhandled error, the framework will reinitialise the consumer and producer and wait for 3 seconds.
Future work:
Features to be added:
- Add decorators for app events
on_consume
,on_processed
andon_publised
. This will allow to hook up e.g. Datadog metrics to these events.
How to deploy a new version
Steps (can maybe be improved):
- change version in setup.py
- git add + commit + push
- create new release in GitHub (copy link)
- update download_url in setup.py
- git add + commit + push (again)
- Run
python setup.py sdist
- Run
twine upload dist/* --verbose
(if not installed,pip install twine
first)
Useful links used:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
kaf-0.1.2b0.tar.gz
(4.3 kB
view hashes)