eventy3 3.0.0

Qotto/eventy

Release 3.0.0

This is the release-3.0.0 branch, will be merged into main tagged 3.0.0. Original README at the bottom

Quickstart

The first feature is a concept of trace_id and request_id context that works with threaded and async execution contexts (using contextvars). This allows to use logging handler defined in eventy to add this information in your logs automatically.

Logging setup API

Out of the box:

import logging

from eventy.logging import GkeHandler, SimpleHandler

# Apply to all loggers
# Log levels can then be reduced in handlers
root_logger = logging.getLogger()
root_logger.setLevel('DEBUG')

# All INFO and above messages with GKE format
gke_handler = GkeHandler(level='INFO')
root_logger.addHandler(gke_handler)

# Additionally, all messages with default colored text format
colortext_handler = SimpleHandler(colored=True, level='DEBUG')
root_logger.addHandler(colortext_handler)

# Additionally, all WARNING with a specific format
custom_handler = SimpleHandler(
    fmt='%(asctime)s SPECIAL WARNING! TRACE_id=%(trace_id)s %(message)s',
    datefmt='[%Y-%m-%d %H:%M:%S %z]',
    level='WARNING'
)
root_logger.addHandler(custom_handler)

Customize everything:

import logging
from eventy.logging import add_trace_id_filter, add_request_id_filter

handler = logging.StreamHandler()
formatter = logging.Formatter(
    fmt='%(asctime)s [CID:%(correlation_id)s] [RID:%(request_id)s] '
        '%(levelname)s (%(module)s L%(lineno)d) '
        '%(message)s',
    datefmt='[%Y-%m-%d %H:%M:%S %z]',
)
handler.setFormatter(formatter)

# You need to add this if you don't use GkeHandler or SimpleHandler
handler.addFilter(add_trace_id_filter)
handler.addFilter(add_request_id_filter)

handler.setLevel('INFO')
root_logger.addHandler(handler)

Context API

Consider this example:

import asyncio
import logging

from eventy.context import request_id
from eventy.logging import SimpleHandler

logger = logging.getLogger(__name__)

root_logger = logging.getLogger()
root_logger.setLevel('DEBUG')
root_logger.addHandler(SimpleHandler())

async def handle_request(delay):
    logger.info(f'Begin')
    await asyncio.sleep(delay)
    logger.info(f'End')

async def main():
    tasks = []

    request_id.set("R-1")
    tasks.append(asyncio.create_task(handle_request(1.0)))
    request_id.set("R-2")
    tasks.append(asyncio.create_task(handle_request(2.0)))
    request_id.set("R-3")
    tasks.append(asyncio.create_task(handle_request(1.5)))

    for task in tasks:
        await task

asyncio.run(main())

Note that the only parameter fo handle_request is the delay to process. The logs are:

[2021-06-22 12:15:08 +0200] [TID:] [RID:R-1] INFO __main__ (ex2 L18) Begin
[2021-06-22 12:15:08 +0200] [TID:] [RID:R-2] INFO __main__ (ex2 L18) Begin
[2021-06-22 12:15:08 +0200] [TID:] [RID:R-3] INFO __main__ (ex2 L18) Begin
[2021-06-22 12:15:09 +0200] [TID:] [RID:R-1] INFO __main__ (ex2 L20) End
[2021-06-22 12:15:09 +0200] [TID:] [RID:R-3] INFO __main__ (ex2 L20) End
[2021-06-22 12:15:10 +0200] [TID:] [RID:R-2] INFO __main__ (ex2 L20) End

Threading support will come later.

Eventy

What is Eventy?

Eventy is both a protocol and a library for making the design of fault-tolerant, event-driven, concurrent and distributed applications in a microservices-oriented architecture easier.

As a protocol, Eventy is language-agnostic.

However, the reference implementation is written in Python. Whatever is your programming language, you can use Eventy. If you are using Python, you will be able to make the most of it quicker.

Motivation

The reality is, a distributed, microservices-oriented architecture is a bit hard to make right.

Martin Fowler even ended up stating:

First Law of Distributed Object Design: “don't distribute your objects”.

He later detailed his view of the First Law regarding microservices.

As Martin Folwer points out, inherently microservices come with their undeniable advantages and a few companies use them with great success. The only reason why a lot of people end up struggling with microservices, is because it greatly increases the complexity of the whole system, which makes it harder to get it right.

Eventy is adressing exactly that issue — providing you with a reliable framework to make microservices work for you.

A big part of the design process to create Eventy was to understand different use cases and see how the design can be simplified. For instance, not only does Eventy offer state partitionning for free, it actually comes up with a few stratagies that eliminate the need to persist state altogether.

Inspiration

Kafka Streams was a great influence in making Eventy. Celery and Faust are also worth to be looked at if you are looking for an opiniated framework easy to get started with.

However, these frameworks only partially solve all the issues you will have with microservices. And, in our opinion, these frameworks are not suitable for designing large critical systems.

They're both opinanated, and therefore cannot be easily integrated in your existing software. You will have to build your software around the framework, instead of the other way around. They also don't give you the full control on the way you can use them: you can only use them as a whole, or not at all.

What Eventy can do for you

Eventy implements multiple features, but all of them simply solve two main problems:

• How to make services communicate with each other
• How to access and persist state

With Eventy, you can serialize data the way you want. You can use Avro, JSON, gRPC, or whatever customer serializer you like.

With Eventy, you can use any system you like as a persistency layer, as long as it supports transactions, if you need strong processing guarantees. The most obvious choice is to use Apache Kafka, but persisting messages over PostgreSQL is completely feasable, too.

Eventy was destined with the mindset of a library of related but independently usable components - and not a framework: the behaviour is explicit and you're the one in charge: you can design your software your own way.

This explicit behaviour, albeit requiring more boilerplate, gives you better clarity on what is happening. Recipes and examples are provided so that you can understand how to use Eventy for most use cases.

You're free to use any part of Eventy as well. Even if you end up not using the Eventy protocol at all, simply reading the documentation and understanding the issues that are adressed and how they are adressed can help you to get on the right path.

Main components of Eventy

• a well-defined communication protocol for sending various types of persisted messages, called Records: Events, Requests and Responses
• persistency of Records that can be stored forever, which lets you keep track of all the changes in your system (especially useful for audits and business analytics)
• queues so that Records can be processed asynchroneously, and aren't lost even if your system is down or overloaded
• strong processing guarantees: a Record can be designed to be processed at least once, at most once and exactly once even if your system encounters a process or network failure at any point
• self-propagating Contexts that in many cases entirely eliminate the need of persisting state
• partitionned state persistency so that you no longer have a single point of failure in your system (aka the database) and can scale up as your business grows