agentops-otel

Shared logging + OpenTelemetry bootstrap for AgentOps.

Logging (configure_logging, get_logger)

One entry point sets up the process's logging; every module then grabs a logger and lets records propagate to the single root handler:

from agentops_otel import configure_logging, get_logger

configure_logging()          # once, at process start (idempotent)
log = get_logger(__name__)   # in every module
log.info("worker started", extra={"worker_id": wid})

configure_logging() installs one stdout handler on the root logger and picks its shape from the environment:

• Local (AGENTOPS_ENV / AGENTOPS_DD_ENV / ENV unset or local) — a readable, coloured console line: time LEVEL name file:line message key=val.
• Anything else (staging, production, …) — one-line JSON per event with explicit level/status and Datadog unified-service dd.* tags, so a Datadog agent tailing container stdout reads the real severity and correlates to traces.

Our staging/prod control plane sets AGENTOPS_DD_ENV, and worker pods set OTEL_EXPORTER_OTLP_ENDPOINT — either signal flips a deployed process to JSON automatically, with no extra config. Overrides: json_logs=True|False forces a format; level comes from the arg, then AGENTOPS_LOG_LEVEL, then AGENTOPS_DEBUG, then INFO.

Trace correlation is generic OpenTelemetry. We read the current span through the OTel API only. When no tracer/exporter is configured (the default — configure_telemetry is opt-in), there is no active span and nothing is added, so logging needs no running collector. When a span is active, each JSON record gains otelTraceID/otelSpanID plus the dd.trace_id/dd.span_id (lower-64-bit) form Datadog uses.

Forwarding worker logs to the control plane (CPForwardingHandler)

agentops_otel.CPForwardingHandler bridges the standard logging module to a sink callback — the SDK worker runtime (komodor_agentops.worker.logging_bridge.install_cp_log_forwarding) uses it to also ship every worker log record to the control plane as a log.created event, since Komodor has no pod/log access to customer-hosted workers. Two env vars control it:

• AGENTOPS_CP_LOG_LEVEL (default INFO) — minimum level forwarded to CP.
• AGENTOPS_CP_LOG_CAPTURE_ALL (default off) — forward only the SDK's own loggers (komodor_agentops, agentops) when unset; set to attach to the root logger instead and capture the whole process.

no_forward is a required keyword argument, and constructing the handler without it is a TypeError by design. Pass the logger prefixes on your delivery path — anything that writes to the control plane, matched hierarchically so a package name covers every module in it:

CPForwardingHandler(sink, no_forward=("myapp.delivery",))   # covers myapp.delivery.*

Forwarding a delivery-path logger's own records creates a feedback loop: the error logged for a rejected event becomes a new event, which is rejected, which logs. It sustains itself at gain 1 — measured at ~95 requests/second per idle worker before it was found. This package deliberately ships no default list, because it cannot know another package's delivery path: it previously defaulted to one hardcoded SDK module name, which both failed to cover the module that actually mattered and stopped matching anything once the SDK renamed it. See the class docstring for the full history, and komodor_agentops/worker/logging_bridge.py for a worked example that derives the set from module __name__s rather than writing them as strings.

Telemetry (configure_telemetry)

configure_telemetry(service_name, *, app=None, engine=None) builds OTLP tracer/meter/logger providers (HTTP/protobuf or gRPC), attaches the OTel logging handler, and runs the FastAPI / httpx / logging auto-instrumentors. SQLAlchemy instrumentation is the optional agentops-otel[sqlalchemy] extra.

Off by default: nothing is installed unless OTEL_EXPORTER_OTLP_ENDPOINT or AGENTOPS_OTEL_ENABLED is set. Used by the control plane and by the SDK worker runtime; neither duplicates the setup.

Metric naming (metric_name, METRIC_PREFIX)

Every AgentOps metric lives under a single agentops. namespace root so the whole product's metrics are isolated and trivially sliceable in Datadog. When you create an instrument, name it through metric_name() rather than passing a raw string:

from agentops_otel import metric_name
from opentelemetry import metrics

meter = metrics.get_meter("agentops.controlplane.authz")
hist = meter.create_histogram(metric_name("authz.pdp.duration_ms"), unit="ms")
# → exported as "agentops.authz.pdp.duration_ms"

metric_name() is idempotent — a name that already starts with METRIC_PREFIX ("agentops.") is returned unchanged, so passing either the bare suffix or the full name is safe.

Why a helper and not a global rewrite? OpenTelemetry Python has no View/exporter-level facility for dynamic per-instrument renaming (a wildcard View's name is a single static string), so the exported metric name is fixed at instrument creation. This helper is therefore the one canonical entry point. Note that the meter/instrumentation-scope name is separate from the metric name and does not affect it — the agentops. prefix must be on the instrument name.

Span attributes (SpanAttr, ProvisionPath)

This package also owns the span-attribute vocabulary — every custom span-tag key AgentOps sets, as one StrEnum. It lives here, beside the bootstrap that emits the spans, rather than in any one service: the keys are a wire contract with Datadog (monitors, dashboards and facets are keyed on these strings), so changing a value breaks dashboards rather than being a rename.

Add a new key here, never as a bare string literal:

from agentops_otel import SpanAttr
from opentelemetry import trace

trace.get_current_span().set_attributes({SpanAttr.ACCOUNT_ID: account_id})
# → exported as "agentops.account.id"

Being a StrEnum, each member is its value — pass it straight to set_attribute() / set_attributes() and compare it as a bare string.

Two namespaces, and only two:

• Reserved usr.* (usr.id/usr.email/usr.name) — Datadog's native user facet, human principals only. usr.id is also the RUM↔APM join key: it equals the UI's RUM setUser id, which is what links a browser session to its backend trace.
• One agentops.* object — everything Datadog has no native concept of (tenant, principal, auth path, impersonation, authz decisions). Dotted keys nest into a single agentops facet tree, so our tags never collide with another instrumentation's.

test_attributes.py enforces both rules mechanically (namespacing, and value uniqueness — a duplicate value silently aliases one member out of the enum, so the tag it was meant to emit is simply never set).

ProvisionPath is the companion value vocabulary for SpanAttr.PROVISION_PATH: how an SSO login resolved to an account. It is set on the span and logged as the path field, so the two can never drift.

The control plane is today's only writer; workers get the vocabulary for free with this package. Enum members carry no dependency of their own — attributes.py imports nothing but enum.