High Level Kafka Scanner, supporting inverse consuming and deduplication. Based on kafka-python library.
Project description
High Level Kafka Scanner
Features:
based on kafka-python library
reverse reading of a kafka topic in batches
deduplication by key
provides fake kafka-python consumer/client for mocking when testing code that uses this library classes
Two classes are provided:
- KafkaScanner - reverse scan feature. Because the particular usage of the inverse logic, this class doesn’t
commit offsets (and so doesn’t support consumer group). It always start from the latest offsets down to the lowest offsets.
KafkaScannerDirect - direct scan.
Check classes docstrings for parameters and more information
Basic example
from kafka_scanner import KafkaScanner
KAFKA_BROKERS = ['kafka1.example.com:9092', 'kafka2.example.com:9092', 'kafka3.example.com:9092']
scanner = KafkaScanner(KAFKA_BROKERS, <topic name>, partitions=[<num partition>])
batches = scanner.scan_topic_batches()
for b in batches:
for m in b:
do_my_thing(m)
SSL example
Set the ssl configs in a dict ssl_configs and pass it to the scanner constructor.
from kafka_scanner import KafkaScanner
KAFKA_BROKERS = ['kafka1.example.com:9093', 'kafka2.example.com:9093', 'kafka3.example.com:9093']
ssl_configs = {
'ssl_cafile': '/path/to/ca.crt',
'ssl_certfile': '/path/to/client.crt',
'ssl_keyfile': '/path/to/client.key',
}
scanner = KafkaScanner(KAFKA_BROKERS, <topic name>, partitions=[<num partition>], ssl_configs=ssl_configs)
...
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for kafka_scanner-0.3.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8604e3c0a740f55f506062b381ae7611f92e5a0c90fe0c4af542584790e3e322 |
|
MD5 | 462e3bf90c7b8b76a89b3af5a75533b2 |
|
BLAKE2b-256 | d9c7c6258bf947c0cb37b91717eb3eb91d4651024373af8018c44556bc7c6e1e |
Hashes for kafka_scanner-0.3.1-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b93d346c91a10b398a47ddf54d0927e6b7a7626e4a52752e08021168651ab10d |
|
MD5 | bfae566086f4b9dc1a0a800e15473a08 |
|
BLAKE2b-256 | a1ef6f1a9cffd7bc124e5b58a0bcdf92a374a83bdcd2c61f7fd14e2ee0dc1bf1 |