AWS Durable Execution SDK - OpenTelemetry Plugin

OpenTelemetry instrumentation plugin for the AWS Durable Execution SDK for Python. Emits durable execution spans on one execution trace, with deterministic workflow, synthetic-root, and operation span IDs.

Features

• Shared Execution Trace: Workflow and Invocation spans share one trace, anchored to a propagated backend parent when available or a deterministic synthetic execution root otherwise
• Same-Trace Ambient Parenting: Invocation spans use the active ambient span only when it already belongs to the execution trace
• Span-per-Operation: Each durable operation (step, wait, invoke) gets its own span with accurate timing
• Continuation Spans: Operations completing in another invocation produce a new correlated span without fabricating an unobserved prior span context
• Log Correlation: Enrich application logs with trace ID and span ID for end-to-end observability
• Provider Integration: Use the global ADOT provider or supply an explicit SDK TracerProvider
• Execution Sampling: Resolve sampling once per invocation and apply it consistently to Workflow, Invocation, operation, and attempt spans

Installation

When using an ADOT or community OpenTelemetry Lambda layer:

pip install aws-durable-execution-sdk-python-otel

The base package intentionally does not install OpenTelemetry libraries. The Lambda layer supplies a version-aligned API, SDK, exporter, and propagators, preventing packages in the function artifact from shadowing parts of the layer.

For an application that configures its own OpenTelemetry provider instead of using a Lambda layer:

pip install "aws-durable-execution-sdk-python-otel[standalone]"

The standalone extra installs the OpenTelemetry API, SDK, and AWS X-Ray propagator. The application remains responsible for configuring its provider, processors, and exporter.

Quick Start using X-Ray/CloudWatch Tracing

1. Add the ADOT Lambda Layer to your function and set AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
2. Enable X-Ray Active Tracing on the function
3. Pass InvocationOtelPlugin to your handler's plugins list
4. Add X-Ray write permissions

Alternatively, install this package in the function artifact or a Lambda layer and select either OTel plugin by entry-point name:

DURABLE_EXECUTION_PLUGINS=otel-invocation
DURABLE_EXECUTION_PLUGINS=otel-execution

otel-invocation creates InvocationOtelPlugin; otel-execution creates ExecutionOtelPlugin. The SDK discovers the selected package entry point at cold start, so the handler does not need to import or explicitly register the plugin.

1. ADOT Lambda Layer

This plugin requires the AWS Distro for OpenTelemetry (ADOT) Lambda layer to export traces from your Lambda function.

The layer ARN follows the format:

arn:aws:lambda:<region>:<awsAccountId>:layer:aws-otel-python-<arch>-ver-<version>

Refer to the ADOT Lambda Layer ARNs page for the latest version number, architecture, and supported regions.

AWS CLI:

aws lambda update-function-configuration \
  --function-name your-function-name \
  --layers "arn:aws:lambda:<region>:<awsAccountId>:layer:aws-otel-python-amd64-ver-<version>"

You must also set the AWS_LAMBDA_EXEC_WRAPPER environment variable:

aws lambda update-function-configuration \
  --function-name your-function-name \
  --environment "Variables={AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument}"

Note: Replace <region> with your function's region and <version>/<arch> with the latest layer version and architecture from the ADOT docs.

CloudFormation / SAM:

MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    Layers:
      - !Sub arn:aws:lambda:${AWS::Region}:<awsAccountId>:layer:aws-otel-python-amd64-ver-<version>
    Environment:
      Variables:
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument

CDK:

from aws_cdk import aws_lambda as lambda_

adot_layer = lambda_.LayerVersion.from_layer_version_arn(
    self,
    "AdotLayer",
    f"arn:aws:lambda:<region>:<awsAccountId>:layer:aws-otel-python-amd64-ver-<version>",
)

fn = lambda_.Function(
    self,
    "MyFunction",
    runtime=lambda_.Runtime.PYTHON_3_12,
    handler="index.handler",
    code=lambda_.Code.from_asset("lambda"),
    layers=[adot_layer],
    environment={"AWS_LAMBDA_EXEC_WRAPPER": "/opt/otel-instrument"},
)

Tip: Pin the layer version to a specific number in production deployments to avoid unexpected behavior from automatic version changes.

2. AWS X-Ray Active Tracing

Enable active tracing on your Lambda function so the _X_AMZN_TRACE_ID environment variable is populated at invocation time. The plugin uses this header to anchor the execution trace on the propagated X-Ray Root/Parent when both are valid, and preserves Sampled=1 or Sampled=0 as the backend sampling decision.

AWS Console: Lambda → Configuration → Monitoring and operations tools → Active tracing → Enable

AWS CLI:

aws lambda update-function-configuration \
  --function-name your-function-name \
  --tracing-config Mode=Active

CloudFormation / SAM:

MyFunction:
  Type: AWS::Lambda::Function
  Properties:
    TracingConfig:
      Mode: Active

CDK:

lambda_.Function(
    self,
    "MyFunction",
    tracing=lambda_.Tracing.ACTIVE,
)

3. In your Lambda handler (index.py)

from aws_durable_execution_sdk_python import DurableContext
from aws_durable_execution_sdk_python.execution import durable_execution
from aws_durable_execution_sdk_python_otel import InvocationOtelPlugin


@durable_execution(plugins=[InvocationOtelPlugin()])
def handler(event: dict, context: DurableContext) -> dict:
    result = context.step(lambda _: fetch_data(event["id"]), name="fetch-data")

    context.wait(duration=Duration.from_seconds(5))

    context.step(lambda _: process(result), name="process")

    return result

The ADOT layer supplies the global TracerProvider; the plugin handles deterministic ID generation and span lifecycle.

4. Grant Permissions

The function's execution role needs the AWSXRayDaemonWriteAccess managed policy (or equivalent permissions) if using X-Ray as the tracing backend.

Environment Variables for ADOT layer

See the ADOT sampling configuration for more details. When the backend header contains an explicit Sampled value, that backend decision takes precedence over local sampler configuration for durable spans.

Configuration

Plugin Options

from aws_durable_execution_sdk_python_otel import (
    InvocationOtelPlugin,
    OtelPluginConfig,
    xray_context_extractor,
)

plugin = InvocationOtelPlugin(
    OtelPluginConfig(
        # Use a custom context extractor (default: xray_context_extractor).
        context_extractor=xray_context_extractor,
        # Custom instrumentation scope name
        # (default: "aws-durable-execution-sdk-python").
        instrument_name="my-service",
        # Install a root-logger filter that stamps trace context onto every
        # log record (default: True).
        enrich_logger=True,
    )
)

Context Extractors

Context extractors return an ExtractedContext object, or None when no durable execution trace context is available. The object carries:

• trace_id: 128-bit OpenTelemetry trace ID
• parent_span_id: 64-bit OpenTelemetry parent span ID
• sampling: Sampling.SAMPLED, Sampling.NOT_SAMPLED, or Sampling.UNDECIDED

The plugin supports multiple strategies for extracting durable execution trace context:

from aws_durable_execution_sdk_python_otel import (
    InvocationOtelPlugin,
    OtelPluginConfig,
    w3c_client_context_extractor,
    xray_context_extractor,
)

# Default: X-Ray trace header (recommended for most Lambda deployments).
InvocationOtelPlugin(OtelPluginConfig(context_extractor=xray_context_extractor))

# W3C Trace Context via clientContext (placeholder for backend propagation support).
InvocationOtelPlugin(OtelPluginConfig(context_extractor=w3c_client_context_extractor))

Custom extractors should return ExtractedContext, not an OpenTelemetry Context.

Trace Structure

Both bundled plugins use the same execution ancestor:

• a propagated backend parent when _X_AMZN_TRACE_ID contains a valid Root and Parent
• otherwise a deterministic, non-recording synthetic root derived from the durable execution ARN

InvocationOtelPlugin keeps durable operation spans under the Invocation span and links operations to Workflow:

Execution ancestor
├── Workflow
└── Invocation
    └── operation
        └── operation attempt 1

ExecutionOtelPlugin keeps operation spans under Workflow and links operations to the current Invocation span:

Execution ancestor
├── Workflow
│   └── operation
│       └── operation attempt 1
└── Invocation

If an ambient Lambda span is active and already has the execution trace ID, the Invocation span uses that ambient span as its parent. Ambient spans on a different trace are ignored for durable parenting so Invocation remains on the execution trace.

Sampling

Sampling is resolved once per invocation and carried to every durable span in that invocation. Precedence is:

1. Sampled=1 or Sampled=0 from _X_AMZN_TRACE_ID
2. a same-trace ambient span's recording/sampled state
3. the configured OpenTelemetry sampler

The resolved decision is applied to Workflow, Invocation, operation, and attempt spans. This avoids independently querying stateful or ratio-based samplers for each durable span in the same invocation.

Log Correlation

When enrich_logger=True (the default), the plugin installs a logging filter on the root logger at invocation start. The filter stamps the active OTel trace context onto every emitted log record using these attributes:

• traceId: 32-char hex trace identifier
• spanId: 16-char hex span identifier
• otelTraceSampled: boolean indicating if the trace is sampled

These attributes are only set when a valid span context is active, so any log formatter or schema must treat the fields as optional.

Verification

After deploying your function with the plugin configured:

1. Invoke your durable function — trigger at least one execution that includes multiple steps or a wait/resume cycle.

2. Check the CloudWatch console — Navigate to CloudWatch → Traces in the AWS Console. You should see an execution trace with:

   • A "Workflow" span exported on the terminal invocation
   • An "Invocation" span per invocation
   • Child spans for each durable operation (named after your step names)
   • All invocations of the same execution grouped under one trace ID

3. Check log correlation — verify that your logs include traceId and spanId fields matching the spans in X-Ray.

4. Confirm sampling — If you set OTEL_TRACES_SAMPLER=traceidratio and OTEL_TRACES_SAMPLER_ARG to a value less than 1.0, verify that only the expected proportion of traces appear.

5. Span links — For operations that span multiple invocations (e.g., after a wait resumes), though span links are set, they are not visualized within the CloudWatch console.

Troubleshooting

API Reference

InvocationOtelPlugin

Invocation-rooted view. Implements DurableInstrumentationPlugin from aws_durable_execution_sdk_python.

InvocationOtelPlugin(
    OtelPluginConfig(
        tracer_provider=None,
        context_extractor=None,
        instrument_name="aws-durable-execution-sdk-python",
        enrich_logger=True,
        workflow_span_name="Workflow",
    )
)

Pass tracer_provider=... when the application owns the OpenTelemetry SDK provider. When omitted, the globally configured provider is used.

ExecutionOtelPlugin

Execution-rooted view. Uses the same execution ancestor and sampling behavior as InvocationOtelPlugin, but parents operation spans under Workflow and links them to Invocation.

DeterministicIdGenerator

A custom OpenTelemetry IdGenerator that produces reproducible trace and span IDs from execution metadata. Exported for advanced use cases.

xray_context_extractor

Default context extractor. Reads the _X_AMZN_TRACE_ID environment variable and returns ExtractedContext containing parsed Root, Parent, and Sampled fields when present.

w3c_client_context_extractor

Alternative context extractor placeholder. Returns None until backend W3C traceparent propagation is supported.

ContextExtractor

Type alias for custom context extractor functions: Callable[[InvocationStartInfo], ExtractedContext | None].

ExtractedContext / Sampling

Structured trace context and sampling decision returned by context extractors.

OtelContextLogFilter / install_log_filter

The logging filter (and its installer) used to stamp trace context onto log records. Installed automatically when enrich_logger=True; exported for manual setups.

Requirements

• Python >= 3.11
• aws-durable-execution-sdk-python >= 2.0.0
• An ADOT/community OpenTelemetry Lambda layer, or the standalone extra

License

Apache-2.0