Skip to main content

Fast python kafka client for asyncio.

Project description

https://travis-ci.com/jmf-mordis/asynckafka.svg?branch=master

Fast python Kafka client for asyncio. Asynckafka is written in Cython on top of Rdkafka as Kafka driver.

The documentation can be found here.

Features

  • Consumer using a balanced group
  • Producer

The library was born as a project to learn Cython, right now it only has the basic features implemented, some of the most important missing features are:

  • Offset management in the consumer
  • Precise partition management

Examples

Simple consumer

How to use a consumer:

consumer = Consumer(
    brokers='localhost:9092',
    topics=['my_topic'],
    group_id='my_group_id',
)
consumer.start()

async for message in consumer:
    print(f"Received message: {message.payload}")

Simple producer

How to use a producer:

producer = Producer(brokers="localhost:9092")
producer.start()
await producer.produce("my_topic", b"my_message")

Benchmark

The test was performed in June of 2018 using a single Kafka broker without replication. The purpose of the benchmark was only to have an idea of the order of magnitude of the library’s performance under these conditions.

Comparison between asynckafka and aiokafka in production and consumption:

Production
https://github.com/jmf-mordis/asynckafka/raw/master/docs/benchmark/graphs/producer.png
Consumption
https://github.com/jmf-mordis/asynckafka/raw/master/docs/benchmark/graphs/consumer.png

The benchmark used for asynckafka is in benchmark directory while the benchmark used for aiokafka is in its own repository, also in the benchmark folder.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

asynckafka-0.2.0.tar.gz (421.4 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page