Skip to main content

Kafka integration with asyncio

Project description

|Build status| |Coverage| |Chat on Gitter|

asyncio client for Kafka

AIOKafkaProducer

AIOKafkaProducer is a high-level, asynchronous message producer.

Example of AIOKafkaProducer usage:

from aiokafka import AIOKafkaProducer
import asyncio

async def send_one():
    producer = AIOKafkaProducer(bootstrap_servers='localhost:9092')
    # Get cluster layout and initial topic/partition leadership information
    await producer.start()
    try:
        # Produce message
        await producer.send_and_wait("my_topic", b"Super message")
    finally:
        # Wait for all pending messages to be delivered or expire.
        await producer.stop()

asyncio.run(send_one())

AIOKafkaConsumer

AIOKafkaConsumer is a high-level, asynchronous message consumer. It interacts with the assigned Kafka Group Coordinator node to allow multiple consumers to load balance consumption of topics (requires kafka >= 0.9.0.0).

Example of AIOKafkaConsumer usage:

from aiokafka import AIOKafkaConsumer
import asyncio

async def consume():
    consumer = AIOKafkaConsumer(
        'my_topic', 'my_other_topic',
        bootstrap_servers='localhost:9092',
        group_id="my-group")
    # Get cluster layout and join group `my-group`
    await consumer.start()
    try:
        # Consume messages
        async for msg in consumer:
            print("consumed: ", msg.topic, msg.partition, msg.offset,
                  msg.key, msg.value, msg.timestamp)
    finally:
        # Will leave consumer group; perform autocommit if enabled.
        await consumer.stop()

asyncio.run(consume())

Running tests

Docker is required to run tests. See https://docs.docker.com/engine/installation for installation notes. Also note, that lz4 compression libraries for python will require python-dev package, or python source header files for compilation on Linux. NOTE: You will also need a valid java installation. It’s required for the keytool utility, used to generate ssh keys for some tests.

Setting up tests requirements (assuming you’re within virtualenv on ubuntu 14.04+):

sudo apt-get install -y libsnappy-dev libzstd-dev libkrb5-dev krb5-user
make setup

Running tests with coverage:

make cov

To run tests with a specific version of Kafka (default one is 1.0.2) use KAFKA_VERSION variable:

make cov KAFKA_VERSION=0.10.2.1

Test running cheatsheat:

  • make test FLAGS="-l -x --ff" - run until 1 failure, rerun failed tests first. Great for cleaning up a lot of errors, say after a big refactor.

  • make test FLAGS="-k consumer" - run only the consumer tests.

  • make test FLAGS="-m 'not ssl'" - run tests excluding ssl.

  • make test FLAGS="--no-pull" - do not try to pull new docker image before test run.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

aiokafka-0.10.0.tar.gz (558.5 kB view hashes)

Uploaded source

Built Distributions

aiokafka-0.10.0-cp312-cp312-win_amd64.whl (355.7 kB view hashes)

Uploaded cp312

aiokafka-0.10.0-cp312-cp312-win32.whl (337.3 kB view hashes)

Uploaded cp312

aiokafka-0.10.0-cp311-cp311-win_amd64.whl (358.1 kB view hashes)

Uploaded cp311

aiokafka-0.10.0-cp311-cp311-win32.whl (338.2 kB view hashes)

Uploaded cp311

aiokafka-0.10.0-cp310-cp310-win_amd64.whl (358.3 kB view hashes)

Uploaded cp310

aiokafka-0.10.0-cp310-cp310-win32.whl (339.2 kB view hashes)

Uploaded cp310

aiokafka-0.10.0-cp39-cp39-win_amd64.whl (359.8 kB view hashes)

Uploaded cp39

aiokafka-0.10.0-cp39-cp39-win32.whl (340.6 kB view hashes)

Uploaded cp39

aiokafka-0.10.0-cp38-cp38-win_amd64.whl (360.5 kB view hashes)

Uploaded cp38

aiokafka-0.10.0-cp38-cp38-win32.whl (340.7 kB view hashes)

Uploaded cp38

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page