pyrkafka 0.2.0

Python bindings for rdkafka Rust library

pyrkafka: Python Wrapper for rdkafka (Rust)

Python Library for Apache Kafka

pyrkafka is a Python wrapper around the rdkafka Rust library. It provides a high-level Pythonic API for producing and consuming Kafka messages, leveraging the performance and safety of Rust via PyO3.

Installation

pip install pyrkafka

Requires Python 3.13+.

Getting Started

Producing Messages

from pyrkafka import PyrKafkaProducer

producer = PyrKafkaProducer("localhost:9092")

# Send a message (partitioned round-robin)
producer.produce("my_topic", b"Hello, Kafka!")

# Send with a key (messages with the same key go to the same partition)
producer.produce_with_key("my_topic", b"Hello, Kafka!", "my_key")

# Flush pending messages (also happens automatically when the producer is dropped)
producer.flush()

Consuming Messages

from pyrkafka import PyrKafkaConsumer

consumer = PyrKafkaConsumer("localhost:9092", "my_topic", "my_group")

for message in consumer:
    print(message.decode())

Custom Configuration

Both producer and consumer accept an optional config dict for additional librdkafka configuration:

producer = PyrKafkaProducer("localhost:9092", config={
    "message.timeout.ms": "5000",
    "compression.type": "zstd",
})

consumer = PyrKafkaConsumer("localhost:9092", "my_topic", "my_group", config={
    "auto.offset.reset": "latest",
    "enable.auto.commit": "false",
})