Skip to main content

Prometheus exporter for AWS ECS Task and Container Metrics

Project description

ecs-container-exporter

AWS ECS side car that exports ECS container level docker stats metrics to Prometheus as well as publish it via Statsd.

Motivation

The default metrics available in AWS ECS are limited, mostly at the task level, across all containers in a task; the container level metrics are not available. In addition, more detailed cgroup metrics are also not available, such as per cpu, and memory usage breakdown into cache, rss, etc.

Luckily AWS exposes the docker stats data via a Task metadata endpoint.

The ecs-container-exporter parses this data, and can expose it to Prometheus or push them via StatsD.

Usage

Install via Pip:

$ pip3 install ecs-container-exporter

or via docker:

$ docker pull raags/ecs-container-exporter

On ECS, add the following json to the task definition:

{
	"name": "ecs-container-exporter",
	"image": "raags/ecs-container-exporter:latest",
	"portMappings": [
	{
		"hostPort": 0,
		"protocol": "tcp",
		"containerPort": 9545
	}
	],
	"command": [],
	"cpu": 256,
	"dockerLabels": {
		"PROMETHEUS_EXPORTER_PORT": "9545"
	}
}

The PROMETHEUS_EXPORTER_PORT label is for ECS discovery via https://github.com/teralytics/prometheus-ecs-discovery

To include or exclude application containers use the INCLUDE or EXCLUDE environment variable. By default ecs-container-exporter and ~internal~ecs~pause (a Fargate internal sidecar) is excluded.

Statsd

Version 2.0.0 add Statsd support with --use-statsd flag or env USE_STATSD. Metrics are emitted with DogStatsd Tag format.

Metrics

The metrics are sampled twice as per the configured interval (default 60s), and than aggregated in this interval. This should be set to the Prometheus scrape interval.

CPU

CPU usage ratio is calculated and scaled as per the applicable container or task cpu limit:

Task Limit Container Limit Task Metric Container Metric
0 0 no scaling no scaling
0 x no scaling scale cpu (can burst above 100%)
x 0 scale as per limit scale as per task limit
x x scale as per limit scale as per container limit (can burst above 100%)

Note that unlike docker stats command and others, CPU usage is not scaled to the number of CPUs. This means a task with 4 CPUs with all 4 having full utilization will show up as 400% in docker stats, but 100% here.

Memory

Memory usage and cache is emitted separately, but the memory usage also includes cache, so subtract cache from it to plot application memory usage specifically.

IO

Network

Network metrics were recently added in Fargate 1.4.0 and ECS agent 1.41.0 onwards.

TODO

[] - Support Non ECS docker host containers

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ecs-container-exporter-2.3.0.tar.gz (12.9 kB view details)

Uploaded Source

Built Distribution

ecs_container_exporter-2.3.0-py3-none-any.whl (14.2 kB view details)

Uploaded Python 3

File details

Details for the file ecs-container-exporter-2.3.0.tar.gz.

File metadata

  • Download URL: ecs-container-exporter-2.3.0.tar.gz
  • Upload date:
  • Size: 12.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.22.0 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.8.10

File hashes

Hashes for ecs-container-exporter-2.3.0.tar.gz
Algorithm Hash digest
SHA256 38bcb3621febba703bb5d7bb4ca0d150f88a8bcb48feba4bc2c2d6343acdbc54
MD5 28072550a7f4950c1c480d93be0d0b48
BLAKE2b-256 6a7d712993f1425618d3e245bd0c21985ddcc60d5308fb544ff82ecdaa6c5a32

See more details on using hashes here.

File details

Details for the file ecs_container_exporter-2.3.0-py3-none-any.whl.

File metadata

  • Download URL: ecs_container_exporter-2.3.0-py3-none-any.whl
  • Upload date:
  • Size: 14.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.22.0 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.8.10

File hashes

Hashes for ecs_container_exporter-2.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 92a599f82cecbd5d6cab90c05686ef445ccc89d71ab89f229f38aae9d9350b10
MD5 4cef4e26a7c2d9664329f639309df243
BLAKE2b-256 fe7f063dc3030aa3ea54158cbe0ef7613aae7492a6d222f619663d8b7273c0de

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page