Skip to main content

Checks messages to avoid reprocessing events.

Project description

Build Status

QuintoAndar Kafka Python Library

QuintoAndar's kafka-python lib wrapper with additional functionalities.

KafkaIdempotentConsumer

A simple wrapper for kafka-python lib that uses redis to check duplicate events.

Configuration

Name Description
group_id The consumer group id
bootstrap_servers The bootstrap servers
redis_host The topic to consume from
redis_port The function that processes the event
idempotent_key Function which extract an unique identifier from the event

See examples

Development

Environment

At the bare minimum you'll need the following for your development environment:

  1. Python 3.6.8

It is strongly recommended to also install and use pyenv:

This tool eases the burden of dealing with virtualenvs and having to activate and deactivate'em by hand. Once you run pyenv local my-project-venv the directory you're in will be bound to the my-project-venv virtual environment and then you will have never to bother again activating the correct venv.

Getting Started

1. Clone the project:

git clone git@github.com:quintoandar/kafka-python.git
cd kafka-python

2. Setup the python environment for the project:

make environment

3. Install dependencies

You can just the make recipe:

make install

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for quintoandar-kafka, version 2.0.0
Filename, size File type Python version Upload date Hashes
Filename, size quintoandar_kafka-2.0.0-py3-none-any.whl (4.9 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size quintoandar_kafka-2.0.0.tar.gz (3.5 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page