ecs_composex 0.3.0

Implement for AWS ECS and Docker Compose what SAM is to Serverless for AWS Lambda

Build your infrastructure and deploy your services to AWS services using docker-compose file format.

• Introduction

• License and documentation

• Blog

• GitHub project

• What is next for ECS ComposeX ?

• Credits

Introduction

Docker Compose has been around for a long while and enabled developers to perform local integration testing between their microservices as well as with other dependencies their application have (i.e. a redis or MySQL server).

However, for developers to translate their docker compose file into an AWS infrastructure can be a lot of work. And for the cloud engineers (or DevOps engineers) it can very quickly become something overwhelming to manage at very large scale to ensure best-practices are in place, for example, ensuring least privileges access from a service to another.

This is where ECS ComposeX comes into play.

Translate Docker services into AWS ECS

First ECS ComposeX translates the services definition in the docker compose file into the ECS definitions to allow the service to run on AWS. It will, doing so, create all the necessary elements to ensure a successful and feature rich deployment into ECS.

Note

ECS ComposeX has been built to allow services to run with Fargate or EC2.

Provision other AWS resources your services need

So you have the definitions of your services and they are running on ECS. But what about these other services that you need for your application to work? DBs, notifications, streams etc. Are you going to run your MySQL server onto ECS too or are you going to want to use AWS RDS? How are you going to define the IAM roles and policies for each service? Access Secrets? Configuration settings?

That is the second focus of ECS ComposeX: defining extra sections in the YAML document of your docker compose file, you can define, for your databases, queues, secrets etc.

ECS ComposeX will parse every single one of these components. These components can exist on their own but what is of interest is to allow the services to access these.

That is where ECS ComposeX will automatically take care of all of that for you.

For services like SQS or SNS, it will create the IAM policies and assign the permissions to your ECS Task Role so the service gets access to these via IAM and STS. Credentials will be available through the metadata endpoint, which your SDK will pick immediately.

For services such as RDS or ElasticCache, it will create the security groups ingress rules as needed, and when applicable, will handle to generate secrets and expose these via ECS Secrets to your services.

Implementing least privileges at the heart of ECS ComposeX

One of the most important value add for a team of Cloud/DevOps engineers who have to look after an environment to use ECS ComposeX is the persistent implementation of best practices:

• All microservices are using different sets of credentials

• All microservices are isolated by default and allowed traffic only when explicitly permitted

• All microservices must be defined as the consumer of a resource (DB, Queue, Table) to be granted access to it.

There have been to many instances of breaches on AWS due to a lack of strict IAM definitions and permissions. Automation can solve that problem and with ECS ComposeX the effort is to constantly abide by the least privileges access principle.

Plug-And-Play

ECS ComposeX allows to create not only the resources your application stack needs, but also the underlying infrastrcuture, for example, your networking layer (VPC, subnets etc.) as well as the compute (using SpotFleet by default).

This is to allow developers to deploy in their development accounts without having to concern themselves with network design and capacity planning.

Note

vpc_network_design

ec2_compute_design

syntax_reference

Note

If you do not need extra AWS resources such as SQS queues to be created as part of these microservices deployments, I would recommend to use AWS ECS CLI which does already a lot of the work for the services. Alternatively, use the AWS CLI v2. It is absolutely smashing-ly awesome and might be just what you need This tool aims to reproduce the original ECS CLI behaviour whilst adding logic for non ECS resources that you want to create in your environment.

Fargate First

However the original deployments and work on this project was done using EC2 instances (using SpotFleet mostly), everything is now implemented to work on AWS Fargate First (2020-06-06).

License and documentation

• Free software: GPLv3+

• Documentation:

  • https://docs.ecs-composex.lambda-my-aws.io

Blog

Follow the news and technical articles on using ECS ComposeX on the Blog

• CICD Pipeline for multiple services on AWS ECS with ECS ComposeX

GitHub project

To follow the progress of ECS ComposeX and raise issues/feature requests, you can go to to the ECS ComposeX Project

What is next for ECS ComposeX ?

• Add more resources supports (DynamoDB tables, SNS Topics).

• Enable definition of service mesh and service discovery

First, move this into a CFN Macro, with a simple root template that would take a few settings in and the URL to the Compose file and render all templates within CFN itself via Lambda. Then, with the newly released CFN Private Registries, mutate this system to have fully integrated to CFN objects which will resolve all this.

Credits

This package would not have been possible without the amazing job done by the AWS CloudFormation team! This package would not have been possible without the amazing community around Troposphere! This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.