weni-eda 0.2.0a8

Python library to simplify Event-Driven Architecture (EDA) with Django and RabbitMQ

Weni EDA

weni-eda is a Python library designed to simplify the use of Event-Driven Architecture (EDA). It provides an interface that seamlessly integrates with the Django framework and RabbitMQ messaging service. The design is scalable and intended to support various integrations in the future.

Features

• Easy integration with Django.
• Support for RabbitMQ.
• Simplified event handling and message dispatching.
• Transport-agnostic Message abstraction, so consumers don't depend on the broker library directly.
• Scalable design to accommodate future integrations with other frameworks and messaging services.

Installation

To install the library, use pip:

pip install weni-eda

Configuration

Django Integration

1. Add weni-eda to your Django project:
   Add weni.eda.django.eda_app to your INSTALLED_APPS in settings.py:

   # settings.py
   INSTALLED_APPS = [
       # ... other installed apps
       'weni.eda.django.eda_app',
   ]
   

2. Environment Variables for weni-eda Configuration

   The following environment variables are used to configure the weni-eda library. Here is a detailed explanation of each variable:

   Variable Name	Examples	Description
   EDA_CONSUMERS_HANDLE	"example.event_driven.handle.handle_consumers"	Specifies the handler module for consumer events.
   EDA_BROKER_HOST	"localhost"	The hostname or IP address of the message broker server.
   EDA_VIRTUAL_HOST	"/"	The virtual host to use when connecting to the broker.
   EDA_BROKER_PORT	5672	The port number on which the message broker is listening.
   EDA_BROKER_USER	"guest"	The username for authenticating with the message broker.
   EDA_BROKER_PASSWORD	"guest"	The password for authenticating with the message broker.

   The following variables are used only when connecting over SSL with weni.eda.django.AMQConnectionParamsFactory (used by brokers such as AmazonMQ):

   Variable Name	Examples	Description
   AMQ_BROKER_HOST	"localhost"	The hostname or IP address of the SSL message broker server.
   AMQ_VIRTUAL_HOST	"/"	The virtual host to use when connecting to the SSL broker.
   AMQ_BROKER_USER	"guest"	The username for authenticating with the SSL message broker.
   AMQ_BROKER_PASSWORD	"guest"	The password for authenticating with the SSL message broker.
   AMQ_BROKER_PORT	5671	The SSL port of the message broker. Defaults to 5671.
   AMQ_BROKER_HEARTBEAT	300	Heartbeat interval (seconds) negotiated with the broker. Defaults to 300.
   AMQ_BROKER_SSL_SERVER_HOSTNAME	"broker.host"	Hostname used for SSL certificate verification (SNI). Defaults to AMQ_BROKER_HOST.

3. Creating your event consumers
   We provide an abstract class that facilitates the consumption of messages. To use it, you need to inherit it and declare the consume method as follows:

   from weni.eda.django.consumers import EDAConsumer
   from weni.eda.messages import Message
   
   
   class ExampleConsumer(EDAConsumer):
       def consume(self, message: Message):
           body = message.json()
           self.ack()
   

   Consumers receive a weni.eda.messages.Message, a transport-agnostic wrapper, so you don't need to import the underlying broker library (amqp). It exposes:

   • message.body: the raw message body (bytes) as delivered by the broker
   • message.json(encoding="utf-8"): parses the body as JSON and returns a dict
   • self.ack(): confirms to the broker that the message can be removed from the queue, which prevents it from being reprocessed
   • message.reject(requeue=False): rejects the message (called automatically if consume raises)

   If consume raises an exception, the message is rejected and the error is logged.

4. Registering your event handlers:
   the EDA_CONSUMERS_HANDLE variable indicates the function that will be called when the consumer starts. this function will be responsible for mapping the messages to their respective consumers. The function must be declared as follows:

   import amqp
   
   from .example_consumer import ExampleConsumer
   
   
   def handle_consumers(channel: amqp.Channel):
       channel.basic_consume("example-queue", callback=ExampleConsumer().handle)
   

   This indicates that any message arriving at the example-queue queue will be dispatched to the ExampleConsumer consumer and will fall into its consume method.

5. Starting to consume the queues
   To start consuming messages from the queue, you need to run the edaconsume command as follows:

   python manage.py edaconsume
   

   From then on, all messages that arrive in the queues where your application is written will be dispatched to their respective consumers.

   By default the consumer connects to the standard broker (no SSL). To connect over SSL on port 5671, pass the --params-class flag with the dotted path to the desired ParamsFactory:

   python manage.py edaconsume   # default broker (no SSL)
   python manage.py edaconsume --params-class "weni.eda.django.AMQConnectionParamsFactory"   # SSL 5671
   

   You can also override the consumers handle and the connection backend per invocation (instead of relying on settings.EDA_CONSUMERS_HANDLE / settings.EDA_CONNECTION_BACKEND). This is useful when one project runs several consumer groups against different brokers (e.g. a buffered SSL consumer alongside the default one):

   python manage.py edaconsume \
       --handle "myapp.messages.handle.handle_consumers" \
       --backend "weni.eda.backends.pyamqp_flush_backend.PyAMQPFlushConnectionBackend" \
       --params-class "weni.eda.django.AMQConnectionParamsFactory"
   

   • --params-class: dotted path to the ParamsFactory (broker/connection params). Defaults to ConnectionParamsFactory.
   • --handle: dotted path to the handle_consumers(channel) function. Defaults to settings.EDA_CONSUMERS_HANDLE.
   • --backend: dotted path to the connection backend. Defaults to settings.EDA_CONNECTION_BACKEND or the built-in PyAMQPConnectionBackend.

Buffered consumers (optional flush backend)

By default the consume loop drains with a short timeout and ticks AMQP heartbeats on each cycle, so idle connections stay alive. Each consumer must ack messages one by one. Some workloads (high-throughput consumers that batch DB writes) need to buffer messages and flush them periodically. For those cases the library ships an optional PyAMQPFlushConnectionBackend.

Both backends use Python logging for connection lifecycle and error messages. Configure handlers in your Django app (or plug in Sentry) as needed.

The flush backend drains with a timeout so it can flush buffered consumers on a fixed interval. py-amqp is single-threaded and not thread-safe, so this is the safe way to do time-based flushing (no extra IO thread).

It is fully opt-in: consumers that do not need buffering keep using the default backend. To use it, your handle_consumers(channel) must register the consumers and return an iterable of "flushable" objects, where each flushable exposes:

• flush(): persist and ack its buffered work;
• flush_interval (optional float): maximum seconds between flushes (defaults to 1.0).

from weni.eda.backends.pyamqp_flush_backend import PyAMQPFlushConnectionBackend
from weni.eda.django.connection_params import AMQConnectionParamsFactory


def handle_consumers(channel):
    consumer = BufferedConsumer()  # exposes flush() and flush_interval
    consumer.setup(channel)        # channel.basic_qos(...) + channel.basic_consume(...)
    return [consumer]


def run():
    params = AMQConnectionParamsFactory.get_params()
    PyAMQPFlushConnectionBackend(handle_consumers).start_consuming(params)

Returning None (or an empty iterable) from handle_consumers disables periodic flushing, making it behave like a plain timed drain loop. The backend can also be selected via settings.EDA_CONNECTION_BACKEND.

License

This project is licensed under the Mozilla Public License 2.0. See the LICENSE file for the full text.