No project description provided
Project description
Django Kafka Consumer
Purpose
Django Kafka Consumer is an utility for consume events from Kafka
Quick start
-
Add
kafka_consumer
to yourINSTALLED_APPS
setting like this:INSTALLED_APPS = [ # ... 'kafka_consumer', ]
-
Run
python manage.py migrate
to create thekafka_consumer
models. -
Add custom subscribers as classes derived from
kafka_consumer.subscribers.base.BaseSubscriber
-
Prepare settings:
KAFKA_HOSTS = ['kafka-host.com:9092'] KAFKA_CONSUMER_TOPICS = { 'topic_key': { 'topic': 'topic name', # no spaces allowed! 'group': 'topic group', 'client': 'client ID', 'subscribers': ( 'path.to.subscriber.Class', ), 'message_processor': 'processor key', # lookup in KAFKA_CONSUMERS_MESSAGE_PROCESSORS 'wait': 0, # optional, indicates how many seconds Kafka will wait to fillup buffer, None or ommited means wait forever 'max_number_of_messages_in_batch': 200, 'consumer_options': { # Overrides options used to create KafkaConsumer 'auto_offset_reset': 'latest', } }, } KAFKA_CONSUMERS_MESSAGE_PROCESSORS = { 'processor key': { 'class': 'path.to.messageprocessor.Class', # Processors init arguments, e.g. 'rsa_private_key_path': 'path/to/private/key' }, } KAFKA_CONSUMER_SSL_SETTINGS = { 'security_protocol': 'SSL', 'ssl_cafile': '/path/to/file/ca.crt', 'ssl_certfile': '/path/to/file/signed.request.crt', 'ssl_keyfile': '/path/to/some/keyfile.key', }
-
To continuously consume events from kafka run:
python manage.py consume --supervised --topic topic_key
License
The Django Kafka Consumer package is licensed under the FreeBSD License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for django-kafka-consumer-2.0.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f0084715d0cb854038c9f4eebcbc8824b1c22d41e945623e64714a48578d019a |
|
MD5 | ab2f598d4e456a722a7c8a7a687ebccf |
|
BLAKE2b-256 | d74e92974b43193d88202b83d21119394cda294d1fa583781cf55da71ee5d499 |
Close
Hashes for django_kafka_consumer-2.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d96d95a845694af58aa35d59d67d7d81127e9331283a22f7a6aefe8423512ccc |
|
MD5 | 9b924f11f7e70ea3cdaa1bb914f75189 |
|
BLAKE2b-256 | 1f798b5d1f23ab79658e6af0736526cd227c03e17714811ad8a60057ef1f89c1 |