otel-messagequeue-exporter 0.1.0

OpenTelemetry span exporters for AWS SQS and Azure Service Bus

OpenTelemetry Trace Exporter

A Python package for exporting OpenTelemetry traces to AWS SQS and Azure Service Bus in OTLP format (Protobuf or JSON).

Features

• AWS SQS Exporter: Export traces to AWS SQS queues
• Azure Service Bus Exporter: Export traces to Azure Service Bus queues
• OTLP Format Support: Both Protobuf (base64 encoded) and JSON formats
• Batch Processing: Efficient batch span processing
• Environment-based Configuration: Flexible configuration for development and production
• Error Handling: Graceful error handling to avoid blocking applications
• Message Size Validation: Automatic validation against cloud provider limits

Installation

Base Installation

pip install otel-messagequeue-exporter

With AWS Support

pip install otel-messagequeue-exporter[aws]

With Azure Support

pip install otel-messagequeue-exporter[azure]

With All Exporters

pip install otel-messagequeue-exporter[all]

Quick Start

AWS SQS Exporter

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource, SERVICE_NAME

from otel_messagequeue_exporter import SQSSpanExporter

# Create resource
resource = Resource.create(attributes={SERVICE_NAME: "my-service"})

# Create tracer provider
provider = TracerProvider(resource=resource)

# Create SQS exporter
exporter = SQSSpanExporter(
    queue_url="https://sqs.us-east-1.amazonaws.com/123456789/traces",
    region_name="us-east-1",
    encoding="otlp_proto",  # or "otlp_json"
)

# Add batch processor
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)

# Set as global tracer provider
trace.set_tracer_provider(provider)

# Use tracing
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my_operation"):
    print("Doing work...")

Azure Service Bus Exporter

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource, SERVICE_NAME

from otel_messagequeue_exporter import AzureServiceBusSpanExporter

# Create resource
resource = Resource.create(attributes={SERVICE_NAME: "my-service"})

# Create tracer provider
provider = TracerProvider(resource=resource)

# Create Azure Service Bus exporter
exporter = AzureServiceBusSpanExporter(
    connection_string="Endpoint=sb://namespace.servicebus.windows.net/;SharedAccessKeyName=...",
    queue_name="traces",
    encoding="otlp_proto",  # or "otlp_json"
)

# Add batch processor
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)

# Set as global tracer provider
trace.set_tracer_provider(provider)

# Use tracing
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my_operation"):
    print("Doing work...")

Configuration

SQSSpanExporter Parameters

• queue_url (str, required): AWS SQS queue URL
• region_name (str, default: "us-east-1"): AWS region name
• encoding (str, default: "otlp_proto"): Message encoding format ("otlp_proto" or "otlp_json")
• aws_access_key_id (str, optional): AWS access key ID (for development)
• aws_secret_access_key (str, optional): AWS secret access key (for development)

AzureServiceBusSpanExporter Parameters

• connection_string (str, required): Azure Service Bus connection string
• queue_name (str, required): Azure Service Bus queue name
• encoding (str, default: "otlp_proto"): Message encoding format ("otlp_proto" or "otlp_json")
• servicebus_namespace (str, optional): Azure Service Bus namespace (for logging)

Environment-based Configuration

The SQS exporter supports environment-based authentication:

import os

# For development (uses explicit credentials)
os.environ["ENV"] = "DEV"
os.environ["SQS_AWS_ACCESS_KEY_ID"] = "your-access-key"
os.environ["SQS_AWS_SECRET_ACCESS_KEY"] = "your-secret-key"

# For production (uses IAM role)
# Don't set ENV="DEV", and the exporter will use default credentials

Advanced Usage

Batch Processor Configuration

from opentelemetry.sdk.trace.export import BatchSpanProcessor

processor = BatchSpanProcessor(
    exporter,
    max_queue_size=2048,  # Maximum queue size
    max_export_batch_size=512,  # Maximum batch size
    schedule_delay_millis=5000,  # Export interval in milliseconds
    export_timeout_millis=30000,  # Export timeout in milliseconds
)

Graceful Shutdown

import atexit

def shutdown_tracing():
    provider.force_flush(timeout_millis=30000)
    provider.shutdown()

atexit.register(shutdown_tracing)

Adding Span Attributes and Events

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("my_operation") as span:
    # Add attributes
    span.set_attribute("user_id", "12345")
    span.set_attribute("operation_type", "example")

    # Add events
    span.add_event("processing_started", {"item_count": 10})

    # Your code here
    print("Doing work...")

    # Add another event
    span.add_event("processing_completed", {"items_processed": 10})

Message Format

OTLP Protobuf Format

When using encoding="otlp_proto", spans are serialized to OTLP protobuf format and base64 encoded before being sent to the message queue. This is the most efficient format and is recommended for production use.

OTLP JSON Format

When using encoding="otlp_json", spans are serialized to OTLP JSON format. This format is human-readable and useful for debugging, but less efficient than protobuf.

Message Size Limits

• AWS SQS: 256KB maximum message size
• Azure Service Bus: 256KB (Standard tier), 1MB (Premium tier)

The exporters automatically check message sizes and log warnings/errors if limits are exceeded.

Examples

See the examples directory for complete examples:

• basic_usage.py: Basic usage of SQS and Azure exporters

Development

Development Installation

# Clone the repository
git clone https://github.com/NeuralgoLyzr/otel-messagequeue-exporter.git
cd otel-messagequeue-exporter

# Install in editable mode with dev dependencies
pip install -e ".[dev]"

Running Tests

pytest

Code Formatting

black src/ tests/

Type Checking

mypy src/

Linting

flake8 src/ tests/

License

MIT License - see LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.