An application that consumes metrics from Kafka messages and store the results into ta postgres db
Project description
Websites Metrics Consumer
Author: Antonio Di Mariano - antonio.dimariano@gmail.com
Description
This service is part of the Websites Monitoring Application, a small distributed application that aims to produce and collect metrics about the availability of one or more websites.
This service is responsible for consuming messages about metrics being produced to an Apache Kafka Avro Topic by a different service. The main action of this service is to store the incoming message into a PostgreSQL database.
Run
If you want to run from the source code, go to the directory websites_metrics_consumer
and run python main
If you want to use it as package (suggested method) install pip3 install websites_metrics_consumer
Then
- Set the ENV variables as show in this README.md
- Then use it this way
def consume_message():
from websites_metrics_consumer.main import start_consumer
start_consumer()
Requirements
- Python >=3.8
Run
python main
Dependencies
- confluent_kafka==1.7.0
- psycopg2==2.9.1
Run test
NOTE: for simplicity I am assuming you have a running local instance of PostgreSQL with the following information
- database='test'
- user='postgresql'
- password='test123'
- host='localhost'
python -m unittest tests/test_db_operations.py
python -m unittest test/test_events_handler_class.py
Service ENV configuration
If you are using a Broker that uses SSL
, like Aiven.com, you need to download the following certificates files and copy them to a folder
- Access Key
- Access Certificate
- CA Certificate
for more information about SSL and Kafka please read https://docs.confluent.io/3.0.0/kafka/ssl.html
Example of your project folder
myprojectname/
|------configuration/
__init__.py
ca.pem
service.cert
service.key
|------venv
|main.py
|requirements.txt
ENV Variable | VALUE | DESCRIPTION |
---|---|---|
brokers | string | Required. The FQDN of the Apache Kafka Brokers. |
schema_registry | string | Required. The FQDN of the Service Registry. |
ssl_ca_location | string | Required. he relative path to your ca.pem |
ssl_certificate_location | string | Required. The relative path to your service.cert |
ssl_key_location | string | Required. The relative path to your service.key |
security_protocol | string | Required. SSL. |
persistence_conf | string | Required. The full string for connecting to your PostgreSQL database. |
group_id | string | Optional. The consumer group id to assign to the consumer(s) . The default value is metrics_consumer |
consumer_topics | string or list | Optional. The Kafka Avro Topic where to consume messages from. The defaul value is websites_metrics |
logging_level | string | Optional. The level of logging to use for the built-in logging package. The default is logging.INFO |
Example of mandatory ENV variables to use
- brokers=kafka-xxx-yyy-abc.com:12026
- schema_registry=https://user:password@kafka-xxx-yyy-abc.com:12029
- ssl_ca_location=configuration/ca.pem
- ssl_certificate_location=configuration/service.cert
- ssl_key_location=configuration/service.key
- security_protocol=SSL
- persistence_conf=postgres://user:password@mydbhost.com:12024/metrics?sslmode=require
If your broker requires SASL authentication, like Confluent Cloud, these are the ENV variables to include
ENV Variable | VALUE | DESCRIPTION |
---|---|---|
brokers | string | Required. The FQDN of the Apache Kafka Brokers. |
schema_registry | string | Required. The FQDN of the Service Registry. |
sasl_username | string | Required. YOUR USERNAME HERE |
sasl_password | string | Required. YOUR PASSWORD HERE |
schema_registry_basic_auth_user_info | string | Required. AUTH HERE |
schema_registry_basic_auth_credentials_source | string | Required. USER_INFO. |
sasl_mechanisms | string | Required. PLAIN. |
security_protocol | string | Required. SASL_SSL. |
persistence_conf | string | Required. The full string for connecting to your PostgreSQL database. |
group_id | string | Optional. The consumer group id to assign to the consumer(s) . The default value is metrics_consumer |
consumer_topics | string or list | Optional. The Kafka Avro Topic where to consume messages from. The defaul value is websites_metrics |
logging_level | string | Optional. The level of logging to use for the built-in logging package. The default is logging.INFO |
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for websites_metrics_consumer-0.0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1117b0f9b3668dd53f84b813a2ab035e994074590df3d488fb724a3dfd464a8 |
|
MD5 | 1ef6bf5705d6ae93135c9ad3db817c95 |
|
BLAKE2b-256 | 4617055d5552dec74da1b8cf651f678f220afa02a89cb5387fed0ae3e30e9ae9 |
Hashes for websites_metrics_consumer-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95435c9b5a36372de15d7d8f4f035f8c6328ca5e8888148f604155e6ca82bd87 |
|
MD5 | ef1b4f377656645ea5f8d6f9169ae2f6 |
|
BLAKE2b-256 | ea823d2c7ea20dedf0564712c774d9cc8349cf952b2243fa88850c7f65a524bb |