Metamorphosis: Somewhat opinionated event streaming

• Author: Jefferson Heard, jheard at teamworks dot com
• Copyright: 2019 Teamworks Inc.
• License: MIT License

Metamorphosis is an event-streaming based microframework that builds on top of Flask, Graphene, Kafka, and Redis. It allows you to define microservices, which consist of a number of mutations (commands), queries, and event consumers.

It chooses simplicity of implementation over abstraction, configurability, and flexibility, making the assumptions that you will be using Kafka or Confluent for your streams, Graphene for your GraphQL, and Redis for your result broker and steward of application state. If you think these technologies suck, this is probably not your microframework. If you think this might be your microframework anyway either because it's late and you're tired of looking or because you think it might have potential, please feel free to provide pull requests implementing abstraction layers around Kafka, Redis, or Flask. I'm really happy to take contributions to this work.

Also, this framework is based on the famous story about a man who turns into a bug. Plainly I identify deeply with the protagonist, as this is roughly the story of my life. Thus you should assume there are bugs in this code, and you should hunt them down and fix them OR choose to completely ignore them like everyone in Kafka's story. The choice is yours entirely, I can't have all the opinions.

How does it work?

Before you start, you should define your service in terms of what commands and queries it accepts, and what events it publishes and subscribes to. At the very least, it publishes an event for every command that it supports, but there may be other events as well. It should also (generally) subscribe at least once to every event generated by a command. Once you have that structure laid out, see the example and work from that. Or try to fly blind using these instructions that I clearly wrote at 10:39pm in a Holiday Inn while putting off sleep. You can tell which method I think will do you more good. Anyway...

1. Instantiate a Microservice object and give it a service name. This service name also serves as your default topic name on Kafka.
2. Define an EventMutation or AsyncEventMutation for each command your microservice handles. Event mutations are synchronous and will poll, waiting on the event to show up in Redis with either an Exception or a result.
3. Define the mutate_event class-method on each mutation, which should authorize and validate the request, then return an instance of the mutation class or some suitable event.
4. Define and register mutation_consumers, which are functions that accept an event and do something with it, such as create an object in a database, create a structured log record, delete spam from your grandmother, etc. Any exception raised by the consumer will be propagated to the caller in the case of a synchronous mutation. Asynchronous mutations will require you to handle exception events in their own way (possibly simply by logging them).
5. Define some queries.
6. Create a flask app.
7. Init the microservice object with it via init_app.
8. Start the GraphQL server using flask run or gunicorn or however you like to run such things
9. Use gregor.py (haha, apropos English-nerd thing, I.R. Smart) to output the default config YAML to a file and modify it and copy it as much as you want to configure your service consumers.
10. Use gregor.py to start up a microservice consumers as a "node" using the config you just created to control how often the processes kill themselves (very Kafkaesque) and how many processes should be up at a given time.

I thought you had opinions?

You mean other than choosing Kafka, Flask, Graphene, and Redis for you? Why yes, yes I did. Or at least I have a way I envision someone using this thing.

Create one GraphQL schema for your entire service-web

Technically this means that your app.py or some such will import ALL of your Microservice objects, init_app() them at once, and then create a Query class that inherits from all the service.Query classes and Mutations class that inherits from all the service.Mutations classes for all services in your application.

Allow the GraphQL server to be started in "manager" mode or regular mode

In manager mode, you would inherit from ONE more Query and Mutations class before you create your schema. This would add the metamorphosis.service_web set of mutations to your microservices. They allow you to dynamically adjust the number of consumer processes on each node, on a consumer by consumer basis and see a map of the nodes, microservices, consumers, and processes that are keeping your little ant-farm going (if you use this to make an ant farm, please send me a video of the working farm and a signed letter of indemnity if you happen to commit ant-genocide due to a bug in the framework).

The important thing about manager-mode, though, is that it's just another set of GraphQL mutations that produce events on a special "service web topic" on kafka which all consumers on all microservice nodes listen to. This means that you don't want it on the public web. All of these accept an authCode parameter, which is a single code that you can store in an ansible vault, a physical vault, your own eidetic memory, or on a post-it note affixed to your SRE chief's massively over-the-top gaming chair.