opentelemetry-exporter-cloudwatch-emf 0.2.2

OpenTelemetry metrics exporter for AWS CloudWatch using Embedded Metric Format (EMF)

OpenTelemetry CloudWatch EMF Exporter

An OpenTelemetry metrics exporter for AWS CloudWatch using Embedded Metric Format (EMF).

This is an unofficial community project. It is not affiliated with, endorsed by, or maintained by Amazon Web Services (AWS), Amazon CloudWatch, or the OpenTelemetry project.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                            Your Application                                  │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   ┌─────────────┐     ┌─────────────┐     ┌─────────────┐                  │
│   │   Counter   │     │    Gauge    │     │  Histogram  │                  │
│   └──────┬──────┘     └──────┬──────┘     └──────┬──────┘                  │
│          │                   │                   │                          │
│          └───────────────────┼───────────────────┘                          │
│                              ▼                                              │
│                    ┌─────────────────┐                                      │
│                    │   MeterProvider │                                      │
│                    └────────┬────────┘                                      │
│                             │                                               │
│                             ▼                                               │
│               ┌─────────────────────────┐                                   │
│               │ PeriodicExportingReader │                                   │
│               └────────────┬────────────┘                                   │
│                            │                                                │
└────────────────────────────┼────────────────────────────────────────────────┘
                             │
                             ▼
              ┌──────────────────────────────┐
              │  CloudWatchEMFExporter       │
              │  ┌────────────────────────┐  │
              │  │ • Metric Conversion    │  │
              │  │ • Unit Mapping         │  │
              │  │ • Dimension Building   │  │
              │  │ • AWS Auto-Detection   │  │
              │  │ • EMF JSON Formatting  │  │
              │  └────────────────────────┘  │
              └──────────────┬───────────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │     stdout      │
                    │  (EMF JSON)     │
                    └────────┬────────┘
                             │
        ┌────────────────────┼────────────────────┐
        │                    │                    │
        ▼                    ▼                    ▼
┌───────────────┐   ┌───────────────┐   ┌───────────────┐
│  AWS Lambda   │   │   ECS/Fargate │   │  EC2 + CW     │
│  (auto logs)  │   │  (awslogs)    │   │    Agent      │
└───────┬───────┘   └───────┬───────┘   └───────┬───────┘
        │                   │                   │
        └───────────────────┼───────────────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │   CloudWatch Logs     │
                │   (EMF Processing)    │
                └───────────┬───────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │  CloudWatch Metrics   │
                │  • Dashboards         │
                │  • Alarms             │
                │  • Percentiles (p99)  │
                └───────────────────────┘

Why This Exporter?

OpenTelemetry and AWS support CloudWatch EMF primarily through Collector or agent workflows. This package fills the smaller collectorless Python SDK use case: emit valid CloudWatch EMF JSON directly from a Python metrics exporter so Lambda, ECS/Fargate, and EC2 workloads can route metrics through stdout or logs.

Approach	Requires Collector	Cold Start Impact	Complexity
ADOT Collector	✅ Yes	High	High
ADOT Lambda Layer	✅ Yes (embedded)	Medium-High	Medium
This Exporter	❌ No	Minimal	Low

Installation

# From PyPI after release
pip install opentelemetry-exporter-cloudwatch-emf

# From this repository before the first PyPI release
pip install git+ssh://[e-mail]/darrenhaas/cw-emf-exporter.git@v0.2.2

Quick Start

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry_exporter_cloudwatch_emf import CloudWatchEMFExporter

# Create exporter
exporter = CloudWatchEMFExporter(namespace="MyApplication")

# Set up meter provider
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=60000)
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)

# Create and use metrics
meter = metrics.get_meter("my-service")
counter = meter.create_counter("requests", unit="1")
counter.add(1, {"endpoint": "/api/users", "method": "GET"})

Documentation

Document	Description
Architecture	System design, data flow, components
Configuration	All options with examples
API Reference	Complete API documentation
Examples	Usage patterns and recipes
Downstream Consumers	Vendored consumers and review process
Release Process	PyPI release checklist
0.2.2 Upgrade Notes	Summary for downstream agents
Project Notice	Unofficial project and trademark notice
Community Gap	Upstream context for collectorless Python EMF export

Features

Metric Types Support

OTel Metric Type	EMF Output	Percentile Support
Counter	Single value	❌
Gauge	Single value	❌
Histogram	Values array	✅ (p50, p99, etc.)
ExponentialHistogram	Values array	✅ (p50, p99, etc.)

AWS Environment Auto-Detection

┌─────────────────────────────────────────────────────────┐
│              AWS Environment Detection                   │
├─────────────────────────────────────────────────────────┤
│                                                         │
│   Check: AWS_LAMBDA_FUNCTION_NAME?                      │
│          │                                              │
│          ├── Yes ──► Lambda Environment                 │
│          │           • function_name                    │
│          │           • function_version                 │
│          │           • region                           │
│          │           • memory_size                      │
│          │                                              │
│          └── No ──► Check: ECS_CONTAINER_METADATA_URI?  │
│                     │                                   │
│                     ├── Yes ──► ECS/Fargate Environment │
│                     │           • aws_ecs=true          │
│                     │           • region                │
│                     │           • cluster/task metadata │
│                     │                                   │
│                     └── No ──► Check: AWS_REGION?       │
│                                │                        │
│                                ├── Yes ──► EC2/Generic  │
│                                │           • region     │
│                                │                        │
│                                └── No ──► No AWS attrs  │
│                                                         │
└─────────────────────────────────────────────────────────┘

Unit Mapping

OpenTelemetry (UCUM)	CloudWatch Unit
s	Seconds
ms	Milliseconds
us	Microseconds
By	Bytes
KiBy	Kilobytes
MiBy	Megabytes
GiBy	Gigabytes
TiBy	Terabytes
By/s	Bytes/Second
1, {request}, {error}	Count
%	Percent

Configuration Summary

CloudWatchEMFExporter(
    # Required
    namespace="MyApp",              # CloudWatch namespace (1-256 chars)

    # Output
    output=sys.stdout,              # Where to write EMF JSON

    # Dimensions
    dimension_keys=["svc", "env"],  # Filter dimensions (None = all)
    max_dimensions=30,              # Max dimensions (CloudWatch limit: 30)

    # Resolution
    storage_resolution=60,          # 1 = high-res, 60 = standard

    # Log Routing (for CloudWatch Agent)
    log_group_name="/aws/myapp",    # Target log group
    log_stream_name="stream-1",     # Target log stream

    # Features
    auto_detect_aws=True,           # Auto-detect Lambda/ECS/EC2
    histogram_as_values=True,       # Enable percentile calculations

    # Advanced
    timestamp_fn=lambda: int(time.time() * 1000),  # Custom timestamp
)

EMF Output Format

{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [{
      "Namespace": "MyApplication",
      "Dimensions": [["service", "endpoint"]],
      "Metrics": [{
        "Name": "request_latency",
        "Unit": "Milliseconds",
        "StorageResolution": 60
      }]
    }]
  },
  "request_latency": [5, 5, 15, 15, 15],
  "service": "api",
  "endpoint": "/users"
}

Histogram values use bucket representatives repeated by bucket count. Large histograms are proportionally sampled to fit EMF's 100-value numeric array limit.

Development

# Install dev dependencies
uv sync --extra dev

# Run tests
uv run pytest

# Lint, type check, and package validation
uv run flake8 src tests
uv run mypy src
uv run black --check src tests
uv build
uv run twine check dist/*

License

Apache 2.0 - See LICENSE