Skip to main content

Full-Featured Pure-Python Kafka Client

Project description

https://travis-ci.org/Parsely/pykafka.svg?branch=master https://coveralls.io/repos/Parsely/pykafka/badge.svg?branch=master

PyKafka

http://i.imgur.com/ztYl4lG.jpg

PyKafka is a cluster-aware Kafka protocol client for python. It includes python implementations of Kafka producers and consumers.

PyKafka’s primary goal is to provide a similar level of abstraction to the JVM Kafka client using idioms familiar to python programmers and exposing the most pythonic API possible.

You can install PyKafka from PyPI with

$ pip install pykafka

Full documentation for PyKafka can be found on readthedocs.

Getting Started

Assuming you have a Kafka instance running on localhost, you can use PyKafka to connect to it.

>>> from pykafka import KafkaClient
>>> client = KafkaClient(hosts="127.0.0.1:9092")

If the cluster you’ve connected to has any topics defined on it, you can list them with:

>>> client.topics
{'my.test': <pykafka.topic.Topic at 0x19bc8c0 (name=my.test)>}
>>> topic = client.topics['my.test']

Once you’ve got a Topic, you can create a Producer for it and start producing messages.

>>> producer = topic.get_producer()
>>> producer.produce(['test message ' + i ** 2 for i in range(4)])

You can also consume messages from this topic using a Consumer instance.

>>> consumer = topic.get_simple_consumer()
>>> for message in consumer:
    if message is not None:
        print message.offset, message.value
0 test message 0
1 test message 1
2 test message 4
3 test message 9

This SimpleConsumer doesn’t scale - if you have two SimpleConsumers consuming the same topic, they will receive duplicate messages. To get around this, you can use the BalancedConsumer.

>>> balanced_consumer = topic.get_balanced_consumer(
    consumer_group='testgroup',
    auto_commit_enable=True,
    zookeeper_connect='myZkClusterNode1.com:2181,myZkClusterNode2.com:2181/myZkChroot'
)

You can have as many BalancedConsumer instances consuming a topic as that topic has partitions. If they are all connected to the same zookeeper instance, they will communicate with it to automatically balance the partitions between themselves.

What happened to Samsa?

This project used to be called samsa. It has been renamed PyKafka and has been fully overhauled to support Kafka 0.8.2. We chose to target 0.8.2 because it’s currently the latest stable version, and the Offset Commit/Fetch API is stabilized.

The Samsa PyPI package will stay up for the foreseeable future and tags for previous versions will always be available in this repo.

Support

If you need help using PyKafka or have found a bug, please open a github issue or use the Google Group.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pykafka-1.1.0.tar.gz (53.5 kB view details)

Uploaded Source

File details

Details for the file pykafka-1.1.0.tar.gz.

File metadata

  • Download URL: pykafka-1.1.0.tar.gz
  • Upload date:
  • Size: 53.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for pykafka-1.1.0.tar.gz
Algorithm Hash digest
SHA256 f7d0a3b9c794966b28d78a677cf698ec49b00adcfbd4b2318f8607c9878a0911
MD5 be4f46764d9521f084144d0ed51e9da3
BLAKE2b-256 919b3b861f2a3edb4acc92b992155faa982053e0da38a5063088e071e9515204

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page