osint-python-test-bed-adapter 1.0.3

Python adapter for Kafka

python-test-bed-adapter

This is the test-bed adapter for Python: it allows you to easily connect Python services to the Apache Kafka test-bed via Python.

The implementation is a wrapper around Pykafka and avro-python3 offering support for:

• AVRO schema's and messages: both key's and values should have a schema as explained here.
• Kafka consumer and producer for the test-bed topics.
• Management
  • Heartbeat (topic: connect-status-heartbeat), so you know which clients are online. Each time the test-bed-adapter is executed, it starts a heartbeat process to notify the its activity to other clients.
  • Configuration (topic: connect-status-configuration), so you can see which topics clients consume and produce.

Installation

You need to install Python 3+.

To run the examples you will need to install the dependencies specified on the filerequirements.txt For that, run pip3 install -r requirements.txt from the project folder.

Examples and usage

• url_producer: creates a message with 4 URLs to RSS feeds on the topic ('system_rss_urls')
• rss_producer: listens to url messages ('system_rss_urls') and produces RSS messages ('system_rss_urls')
• rss_consumer: listens to RSS messages ('system_rss_urls') and prints them to console.