PyKafka

PyKafka is a cluster-aware Kafka protocol client for python. It includes python implementations of Kafka producers and consumers.

PyKafka’s primary goal is to provide a similar level of abstraction to the JVM Kafka client using idioms familiar to python programmers and exposing the most pythonic API possible.

You can install PyKafka from PyPI with

$ pip install pykafka

Full documentation for PyKafka can be found on readthedocs.

Getting Started

Assuming you have a Kafka instance running on localhost, you can use PyKafka to connect to it.

>>> from pykafka import KafkaClient
>>> client = KafkaClient(hosts="127.0.0.1:9092")

If the cluster you’ve connected to has any topics defined on it, you can list them with:

>>> client.topics
{'my.test': <pykafka.topic.Topic at 0x19bc8c0 (name=my.test)>}
>>> topic = client.topics['my.test']

Once you’ve got a Topic, you can create a Producer for it and start producing messages.

>>> producer = topic.get_producer()
>>> producer.produce(['test message ' + i ** 2 for i in range(4)])

You can also consume messages from this topic using a Consumer instance.

>>> consumer = topic.get_simple_consumer()
>>> for message in consumer:
    if message is not None:
        print message.offset, message.value
0 test message 0
1 test message 1
2 test message 4
3 test message 9

This SimpleConsumer doesn’t scale - if you have two SimpleConsumers consuming the same topic, they will receive duplicate messages. To get around this, you can use the BalancedConsumer.

>>> balanced_consumer = topic.get_balanced_consumer(
        consumer_group='testgroup', auto_commit_enable=True)

You can have as many BalancedConsumer instances consuming a topic as that topic has partitions. If they are all connected to the same zookeeper instance, they will communicate with it to automatically balance the partitions between themselves.

What happened to Samsa?

This project used to be called samsa. It has been renamed PyKafka and has been fully overhauled to support Kafka 0.8.2. We chose to target 0.8.2 because it’s currently the latest stable version, and the Offset Commit/Fetch API is stabilized.

The Samsa PyPI package will stay up for the foreseeable future and tags for previous versions will always be available in this repo.

Support

If you need help using PyKafka or have found a bug, please open a github issue or use the Google Group.

Release files for pykafka 1.0.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pykafka 1.0.3
File	Size	Uploaded
pykafka-1.0.3.tar.gz	49.9 kB	Jun 8, 2015	Details

Release files / pykafka-1.0.3.tar.gz

Download URL	pykafka-1.0.3.tar.gz
Size	49.9 kB
Tags	Source
SHA-256 checksum How to use checksums	`e8247fdf0587784bee5f5f432ab5390c7b029b27f7b13815de5e8695bca11dda`
BLAKE2b-256 checksum How to use checksums	`8a2e56b3aaca319943c86414f185708b638be4d18ee47110f7791d09af413d82`
Upload date	Jun 8, 2015
Uploaded using Trusted Publishing? What is trusted publishing?	No