Skip to main content

Smyte fork of Confluent's Apache Kafka client for Python

Project description

Smyte fork

We created a fork to get timestamps working. This will be removed as soon as our
[pull request](
has been merged into the mainline

Confluent's Apache Kafka client for Python

Confluent's Kafka client for Python wraps the librdkafka C library, providing
full Kafka protocol support with great performance and reliability.

The Python bindings provides a high-level Producer and Consumer with support
for the balanced consumer groups of Apache Kafka 0.9.

See the [API documentation]( for more info.

**License**: [Apache License v2.0](



from confluent_kafka_smyte import Producer

p = Producer({'bootstrap.servers': 'mybroker,mybroker2'})
for data in some_data_source:
p.produce('mytopic', data.encode('utf-8'))

**High-level Consumer:**

from confluent_kafka_smyte import Consumer, KafkaError

c = Consumer({'bootstrap.servers': 'mybroker', '': 'mygroup',
'default.topic.config': {'auto.offset.reset': 'smallest'}})
running = True
while running:
msg = c.poll()
if not msg.error():
print('Received message: %s' % msg.value().decode('utf-8'))
elif msg.error().code() != KafkaError._PARTITION_EOF:
running = False

See [examples](examples) for more examples.

Broker compatibility
The Python client (as well as the underlying C library librdkafka) supports
all broker versions >= 0.8.
But due to the nature of the Kafka protocol in broker versions 0.8 and 0.9 it
is not safe for a client to assume what protocol version is actually supported
by the broker, thus you will need to hint the Python client what protocol
version it may use. This is done through two configuration settings:

* `broker.version.fallback=YOUR_BROKER_VERSION` (default
* `api.version.request=true|false` (default false)

When using a Kafka 0.10 broker or later you only need to set
If you use Kafka broker 0.9 or 0.8 you should leave
`api.version.request=false` (default) and set
`broker.version.fallback` to your broker version,
e.g `broker.version.fallback=`.

More info here:


* Python >= 2.7 or Python 3.x
* [librdkafka]( >= 0.9.1


**Install from PyPi:**

$ pip install confluent-kafka

**Install from source / tarball:**

$ pip install .


$ python build

If librdkafka is installed in a non-standard location provide the include and library directories with:

$ C_INCLUDE_PATH=/path/to/include LIBRARY_PATH=/path/to/lib python ...


**Run unit-tests:**

$ py.test

**NOTE**: Requires `py.test`, install by `pip install pytest`

**Run integration tests:**

$ examples/ <kafka-broker>

**WARNING**: These tests require an active Kafka cluster and will make use of a topic named 'test'.

Generate documentation
Install sphinx and sphinx_rtd_theme packages and then:

$ make docs


$ python build_sphinx

Documentation will be generated in `docs/_build/`.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for confluent-kafka-smyte, version 0.9.3
Filename, size File type Python version Upload date Hashes
Filename, size confluent-kafka-smyte-0.9.3.tar.gz (27.2 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page