comprehend-telemetry 0.2.3

Integration of comprehend.dev with OpenTelemetry in Python

comprehend-telemetry

OpenTelemetry integration for comprehend.dev - automatically capture and analyze your Python application's architecture, performance, and runtime metrics.

Installation

pip install comprehend-telemetry

Quick Start

Starting from scratch

If you don't have OpenTelemetry set up yet, here's a complete setup:

pip install comprehend-telemetry opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.auto_instrumentation import sitecustomize
from comprehend_telemetry import ComprehendSDK

# Set up OpenTelemetry with your service information
resource = Resource.create({
    "service.name": "my-python-service",
    "service.namespace": "production",
    "deployment.environment": "prod"
})

comprehend = ComprehendSDK(
    organization='your-org',
    token=os.getenv("COMPREHEND_SDK_TOKEN"),
    debug=True  # Optional: enable debug logging (or pass a custom logger function)
)

tracer_provider = TracerProvider(
    resource=resource,
    active_span_processor=comprehend.get_span_processor(),
)
trace.set_tracer_provider(tracer_provider)

meter_provider = MeterProvider(
    resource=resource,
    metric_readers=[
        PeriodicExportingMetricReader(
            comprehend.get_metrics_exporter(),
            export_interval_millis=15000,
        )
    ],
)

Adding to existing OpenTelemetry setup

If you already have OpenTelemetry configured, create a ComprehendSDK instance and add its processor and exporter:

import os
from comprehend_telemetry import ComprehendSDK

comprehend = ComprehendSDK(
    organization='your-org',
    token=os.getenv("COMPREHEND_SDK_TOKEN"),
)

# Add to your existing tracer provider:
tracer_provider.add_span_processor(comprehend.get_span_processor())

# Add metrics reader to your meter provider:
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader

metric_reader = PeriodicExportingMetricReader(
    comprehend.get_metrics_exporter(),
    export_interval_millis=15000,
)

Process metrics (CPU and memory)

For process-level CPU and memory metrics, add opentelemetry-instrumentation-system-metrics:

pip install opentelemetry-instrumentation-system-metrics

from opentelemetry.instrumentation.system_metrics import (
    SystemMetricsInstrumentor,
    _DEFAULT_CONFIG,
)

# Disable everything in the default config, then re-enable only what we want.
# A partial config is not sufficient: the instrumentor registers callbacks for
# all metrics in _DEFAULT_CONFIG and looks them up at collection time, so any
# missing key causes a KeyError.
config = {k: [] for k in _DEFAULT_CONFIG}
config.update({
    "process.cpu.time": ["user", "system"],
    "process.cpu.utilization": None,
    "process.memory.usage": None,
    "process.memory.virtual": None,
})
SystemMetricsInstrumentor(config=config).instrument()

This collects process.cpu.time, process.cpu.utilization, process.memory.usage, and process.memory.virtual, while disabling system-wide CPU, memory, disk, and network metrics to avoid unnecessary overhead.

Service instance identity

Set service.instance.id in your resource to give each running process a unique identity that changes on every restart. This lets comprehend.dev distinguish between different instances of the same service and track restarts over time:

import uuid
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "my-python-service",
    "service.instance.id": str(uuid.uuid4()),
})

There is no automatic detector for service.instance.id in the Python OTel SDK (unlike Node.js, which has serviceInstanceIdDetector), so generating a UUID at startup is the recommended approach.

Kubernetes resources

For k8s identity attributes that cannot be read from the host (pod name, namespace, node), use the Kubernetes Downward API to inject them as OTEL_RESOURCE_ATTRIBUTES:

env:
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(POD_NAMESPACE),k8s.node.name=$(NODE_NAME)
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

Configuration

Set your comprehend.dev SDK token as an environment variable:

export COMPREHEND_SDK_TOKEN=your-token-here

Note: In production environments, the token should be stored in a secure secret management system (such as AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, Kubernetes Secrets, or your cloud provider's secret management service) and injected into the environment through your container orchestrator's workload definition or service configuration.

What it captures

This integration automatically captures:

• HTTP Routes - API endpoints and their usage patterns
• Database Operations - SQL queries (analysis done server-side)
• Service Dependencies - HTTP client calls to external services
• Performance Metrics - Request durations, response codes, error rates
• Service Architecture - Automatically maps your service relationships
• Trace Spans - Span identity and parent relationships for connecting observations to traces
• Runtime Metrics - Process CPU and memory metrics
• Custom Metrics - Server-configured custom metric and span collection

Requirements

• Python 3.8+
• OpenTelemetry SDK (peer dependencies: opentelemetry-api, opentelemetry-sdk)

Framework Support

Works with any Python framework that supports OpenTelemetry auto-instrumentation:

• FastAPI
• Django
• Flask
• SQLAlchemy
• Requests
• HTTPx
• psycopg2
• And more...

Learn More

• Visit comprehend.dev for documentation and to get your ingestion token
• OpenTelemetry Python Documentation

Development

See DEVELOPMENT.md for development setup and release instructions.