weni-eda 0.2.0a5

No project description provided

Weni EDA

weni-eda is a Python library designed to simplify the use of Event-Driven Architecture (EDA). It provides an interface that seamlessly integrates with the Django framework and RabbitMQ messaging service. The design is scalable and intended to support various integrations in the future.

Features

• Easy integration with Django.
• Support for RabbitMQ.
• Simplified event handling and message dispatching.
• Scalable design to accommodate future integrations with other frameworks and messaging services.

Installation

To install the library, use pip:

pip install weni-eda

Configuration

Django Integration

1. Add weni-eda to your Django project:
   Add weni.eda.django.eda_app to your INSTALLED_APPS in settings.py:

   # settings.py
   INSTALLED_APPS = [
       # ... other installed apps
       'weni.eda.django.eda_app',
   ]
   

2. Environment Variables for weni-eda Configuration

   The following environment variables are used to configure the weni-eda library. Here is a detailed explanation of each variable:

   Variable Name	Examples	Description
   EDA_CONSUMERS_HANDLE	"example.event_driven.handle.handle_consumers"	Specifies the handler module for consumer events.
   EDA_BROKER_HOST	"localhost"	The hostname or IP address of the message broker server.
   EDA_VIRTUAL_HOST	"/"	The virtual host to use when connecting to the broker.
   EDA_BROKER_PORT	5672	The port number on which the message broker is listening.
   EDA_BROKER_USER	"guest"	The username for authenticating with the message broker.
   EDA_BROKER_PASSWORD	"guest"	The password for authenticating with the message broker.

   The following variables are used only when connecting over SSL with weni.eda.django.AMQConnectionParamsFactory (used by brokers such as AmazonMQ):

   Variable Name	Examples	Description
   AMQ_BROKER_HOST	"localhost"	The hostname or IP address of the SSL message broker server.
   AMQ_VIRTUAL_HOST	"/"	The virtual host to use when connecting to the SSL broker.
   AMQ_BROKER_USER	"guest"	The username for authenticating with the SSL message broker.
   AMQ_BROKER_PASSWORD	"guest"	The password for authenticating with the SSL message broker.
   AMQ_BROKER_PORT	5671	The SSL port of the message broker. Defaults to 5671.
   AMQ_BROKER_HEARTBEAT	300	Heartbeat interval (seconds) negotiated with the broker. Defaults to 300.
   AMQ_BROKER_SSL_SERVER_HOSTNAME	"broker.host"	Hostname used for SSL certificate verification (SNI). Defaults to AMQ_BROKER_HOST.

3. Creating your event consumers
   We provide an abstract class that facilitates the consumption of messages. To use it, you need to inherit it and declare the consume method as follows:

   from weni.eda.django.consumers import EDAConsumer
   
   
   class ExampleConsumer(EDAConsumer):
       def consume(self, message: Message):
           body = JSONParser.parse(message.body)
           self.ack()
   

   • JSONParser.parse(message.body) Converts the message arriving from RabbitMQ in JSON format to dict
   • self.ack() Confirms to RabbitMQ that the message can be removed from the queue, which prevents it from being reprocessed.

4. Registering your event handlers:
   the EDA_CONSUMERS_HANDLE variable indicates the function that will be called when the consumer starts. this function will be responsible for mapping the messages to their respective consumers. The function must be declared as follows:

   import amqp
   
   from .example_consumer import ExampleConsumer
   
   
   def handle_consumers(channel: amqp.Channel):
       channel.basic_consume("example-queue", callback=ExampleConsumer().handle)
   

   This indicates that any message arriving at the example-queue queue will be dispatched to the ExampleConsumer consumer and will fall into its consume method.

5. Starting to consume the queues
   To start consuming messages from the queue, you need to run the edaconsume command as follows:

   python manage.py edaconsume
   

   From then on, all messages that arrive in the queues where your application is written will be dispatched to their respective consumers.

   By default the consumer connects to the standard broker (no SSL). To connect over SSL on port 5671, pass the --params-class flag with the dotted path to the desired ParamsFactory:

   python manage.py edaconsume   # default broker (no SSL)
   python manage.py edaconsume --params-class "weni.eda.django.AMQConnectionParamsFactory"   # SSL 5671
   

Buffered consumers (optional flush backend)

By default the consume loop blocks indefinitely waiting for events, so each consumer must ack messages one by one. Some workloads (high-throughput consumers that batch DB writes) need to buffer messages and flush them periodically. For those cases the library ships an optional PyAMQPFlushConnectionBackend.

This backend drains with a timeout so it can flush buffered consumers on a fixed interval. py-amqp is single-threaded and not thread-safe, so this is the safe way to do time-based flushing (no extra IO thread).

It is fully opt-in: consumers that do not need buffering keep using the default backend. To use it, your handle_consumers(channel) must register the consumers and return an iterable of "flushable" objects, where each flushable exposes:

• flush(): persist and ack its buffered work;
• flush_interval (optional float): maximum seconds between flushes (defaults to 1.0).

from weni.eda.backends.pyamqp_flush_backend import PyAMQPFlushConnectionBackend
from weni.eda.django.connection_params import AMQConnectionParamsFactory


def handle_consumers(channel):
    consumer = BufferedConsumer()  # exposes flush() and flush_interval
    consumer.setup(channel)        # channel.basic_qos(...) + channel.basic_consume(...)
    return [consumer]


def run():
    params = AMQConnectionParamsFactory.get_params()
    PyAMQPFlushConnectionBackend(handle_consumers).start_consuming(params)

Returning None (or an empty iterable) from handle_consumers disables periodic flushing, making it behave like a plain timed drain loop. The backend can also be selected via settings.EDA_CONNECTION_BACKEND.