ol-openedx-otel-monitoring 0.2.0

OTel Monitoring

This Django plugin integrates Open Telemetry, offering both tracing and metric functionalities. It is built upon the opentelemetry-instrumentation-django package and adds the capability for manual instrumentation. Users can select different exporters for traces and metrics. Currently, it supports Console Exporter for local development and OTLP Exporter for exporting data to third-party platforms. A customizable blueprint middleware can be easily added via settings.

Installation

For detailed installation instructions, please refer to the plugin installation guide.

Installation required in:

• LMS

• CMS

Configuration

The plugin offers multiple settings for customization that can be set using the config file in Open edX. For any release after Juniper, that config file is /edx/etc/lms.yml. If you’re using private.py, add these values to lms/envs/private.py.

OTEL_CONFIGS = {
    "OTEL_ENABLED": True,
    "OTEL_TRACES_ENABLED": True,
    "OTEL_METRICS_ENABLED": True,
    "TRACES_EXPORTER": "console",
    "METRICS_EXPORTER": "console",
    "OTEL_INSTRUMENTATION_SQLCOMMENTER_ENABLED": False,
}

OTEL_CONFIGS allows the following settings:

• OTEL_ENABLED: Enables or disables telemetry data export.

• OTEL_TRACES_ENABLED: Enables or disables traces telemetry data export.

• OTEL_METRICS_ENABLED: Enables or disables metrics telemetry data export.

• TRACES_EXPORTER: Defines the exporter for traces.

• METRICS_EXPORTER: Specifies the exporter for metrics.

• OTEL_INSTRUMENTATION_SQLCOMMENTER_ENABLED: Toggles SQLCommenter for Django instrumentation.

Exporter mappings:

The plugin allows dynamic selection and configuration of exporters for both traces and metrics:

OTEL_TRACES_EXPORTER_MAPPING = {
    "console": "opentelemetry.sdk.trace.export.ConsoleSpanExporter",
    "otlphttp": "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter",
}

OTEL_METRICS_EXPORTER_MAPPING = {
    "console": "opentelemetry.sdk.metrics.export.ConsoleMetricExporter",
    "otlphttp": "opentelemetry.exporter.otlp.proto.http.metric_exporter.OTLPMetricExporter",
}

To add a new exporter, include it in the appropriate mapping and select it in the configuration. Support for the new exporter must also be implemented in the plugin. Any additional settings required for the exporter should be added to the settings for easy access within the plugin.

Resource Attributes:

OTEL_TRACES_RESOURCE_ATTRIBUTE

This setting allows you to specify the resource attributes that should be associated with traces. It enhances trace data by providing additional context, making it easier to filter and analyze trace information.

OTEL_TRACES_RESOURCE_ATTRIBUTE = {
    'service.name': 'example-service',
    'service.instance.id': 'instance-1'
    # Other attributes can be added here
}

OTEL_METRICS_RESOURCE_ATTRIBUTE

Similarly, it is used for specifying resource attributes for metrics. It helps in enriching metric data with useful information, aiding in the monitoring and analysis process.

OTEL_METRICS_RESOURCE_ATTRIBUTE = {
    'service.name': 'example-service',
    'service.instance.id': 'instance-1'
    # Additional attributes can be included here
}

For more detailed information about these settings and how to use them, please refer to the following documentation: Resource semantic conventions & Resource SDK

OTLP exporter configuration:

It is recommended to use separate settings for traces and metrics rather than generic settings. For example:

OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4318"

In this case, traces will automatically export to <OTEL_EXPORTER_OTLP_ENDPOINT>/v1/traces and metrics to <OTEL_EXPORTER_OTLP_ENDPOINT>/v1/metrics. However, if you require separate configurations for traces and metrics, you can use:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = "http://localhost:4318/v1/traces"
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT = "http://localhost:4318/v1/metrics"

OTEL_EXPORTER_OTLP_TRACES_HEADERS = '{"Authorization": "Basic <base64_encoded_string>"}'
OTEL_EXPORTER_OTLP_METRICS_HEADERS = '{"Authorization": "Basic <base64_encoded_string>"}'
OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE = True
OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE = True
OTEL_EXPORTER_OTLP_TRACES_TIMEOUT = 10
OTEL_EXPORTER_OTLP_METRICS_TIMEOUT = 10
OTEL_EXPORTER_OTLP_TRACES_COMPRESSION = "none"
OTEL_EXPORTER_OTLP_METRICS_COMPRESSION = "none"

This approach allows more flexibility and control over where each type of telemetry data is sent, especially useful in complex deployment environments.

As the plugin currently supports only OTLP HTTP, you need to specify the protocol:

OTEL_EXPORTER_OTLP_TRACES_PROTOCOL = "http/protobuf"

Future updates may include support for other protocols like gRPC.

Settings related to Django instrumentation:

To exclude certain URLs from tracking
OTEL_PYTHON_DJANGO_EXCLUDED_URLS = "healthcheck"

# To extract attributes from Django's request object
OTEL_PYTHON_DJANGO_TRACED_REQUEST_ATTRS = "path_info,content_type"

# To capture HTTP request headers as span attributes
# e.g. content-type,custom_request_header,Accept.*,X-.*,.*
OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST = ".*"

# To capture HTTP response headers as span attributes,
# e.g. content-type,custom_response_header,Content.*,X-.*,.*
OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE = ".*"

# To prevent storing sensitive data e.g. .*session.*,set-cookie
OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS = ".*session.*,set-cookie"

Configure SQLCommenter settings:

# Enabling this flag will add django framework, and it's version
SQLCOMMENTER_WITH_FRAMEWORK = True

# Enabling this flag will add controller name that handles the request
SQLCOMMENTER_WITH_CONTROLLER = True

# Enabling this flag will add url path that handles the request
SQLCOMMENTER_WITH_ROUTE = True

# Enabling this flag will add app name that handles the request
SQLCOMMENTER_WITH_APP_NAME = True

# Enabling this flag will add open-telemetry transparent
SQLCOMMENTER_WITH_OPENTELEMETRY = True

# Enabling this flag will add name of the db driver
SQLCOMMENTER_WITH_DB_DRIVER = True

In addition to Django, this plugin requires several OpenTelemetry-related packages ensure that the following packages are installed.

• opentelemetry-api

• opentelemetry-sdk

• opentelemetry-instrumentation-django

• opentelemetry-exporter-richconsole

• opentelemetry-exporter-otlp-proto-http

How To Use

1. Install the Plugin: Begin by installing the Open Telemetry Django plugin in your Django project.

2. Configure Necessary Settings: Ensure all necessary settings are properly configured. This includes specifying the correct endpoints, exporters, and any Django or SQLCommenter specific settings as outlined in the previous sections.

3. Test the Health Check Endpoint: After installation and configuration, test the plugin’s functionality by hitting the /otel/healthcheck/ endpoint. This can be done using a browser or a tool like curl:

   curl http://localhost:8000/otel/healthcheck/

   Replace localhost:8000 with your actual server address. A successful hit to this endpoint will return a response, confirming that the plugin is healthy.

4. Verify Traces: Upon accessing the health check endpoint, you should be able to see the traces of this request in your configured trace exporter (e.g., Console or OTLP Exporter). This verifies that the plugin is not only installed but also actively tracing requests.

Customizing Traces and Metrics

To add custom data to traces and metrics, enhancing the utility of the telemetry data.

Custom Traces

To add custom data to your traces, use the Open Telemetry tracing API. Here’s a basic example:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("custom_span") as span:
    span.set_attribute("custom_attribute", "value")
    # Your custom code goes here

For more advanced tracing techniques and examples, refer to the detailed tracing guide.

Custom Metrics

Similarly, for metrics, utilize the Open Telemetry metrics API. Below is a simple example:

from opentelemetry import metrics

meter = metrics.get_meter(__name__)
custom_counter = meter.create_counter("custom_counter", description="Custom metric counter")

def some_function():
    custom_counter.add(1)
    # Additional logic for your function

Explore more about metrics instrumentation in the comprehensive metrics guide

Refer to the Open Telemetry Documentation for more details and advanced usage instructions.