Skip to main content

High Level Kafka Scanner, supporting inverse consuming and deduplication. Based on kafka-python library.

Project description

High Level Kafka Scanner

Features:

  • based on kafka-python library

  • reverse reading of a kafka topic in batches

  • deduplication by key

  • provides fake kafka-python consumer/client for mocking when testing code that uses this library classes

Two classes are provided:

  • KafkaScanner - reverse scan feature. Because the particular usage of the inverse logic, this class doesn’t

    commit offsets (and so doesn’t support consumer group). It always start from the latest offsets down to the lowest offsets.

  • KafkaScannerDirect - direct scan.

Check classes docstrings for parameters and more information

Basic example

from kafka_scanner import KafkaScanner
KAFKA_BROKERS = ['kafka1.example.com:9092', 'kafka2.example.com:9092', 'kafka3.example.com:9092']

scanner = KafkaScanner(KAFKA_BROKERS, <topic name>, partitions=[<num partition>])
batches = scanner.scan_topic_batches()
for b in batches:
    for m in b:
        do_my_thing(m)

SSL example

Set the ssl configs in a dict ssl_configs and pass it to the scanner constructor.

from kafka_scanner import KafkaScanner
KAFKA_BROKERS = ['kafka1.example.com:9093', 'kafka2.example.com:9093', 'kafka3.example.com:9093']

ssl_configs = {
    'ssl_cafile': '/path/to/ca.crt',
    'ssl_certfile': '/path/to/client.crt',
    'ssl_keyfile': '/path/to/client.key',
}
scanner = KafkaScanner(KAFKA_BROKERS, <topic name>, partitions=[<num partition>], ssl_configs=ssl_configs)
...

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kafka-scanner-0.3.1.tar.gz (10.6 kB view hashes)

Uploaded source

Built Distributions

kafka_scanner-0.3.1-py3.4.egg (28.7 kB view hashes)

Uploaded 3 4

kafka_scanner-0.3.1-py3-none-any.whl (12.4 kB view hashes)

Uploaded 3 4

kafka_scanner-0.3.1-py2.7.egg (27.9 kB view hashes)

Uploaded 2 7

kafka_scanner-0.3.1-py2-none-any.whl (14.9 kB view hashes)

Uploaded 2 7

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page