Key Components

• TelemetryHandler – manages LLM invocation lifecycles (spans, metrics, events)

• InferenceInvocation and message types (TextPart, ReasoningPart, BlobPart, etc.) – structured data model for GenAI interactions

• CompletionHook – protocol for uploading content to external storage (built-in fsspec support)

• Metrics – gen_ai.client.operation.duration and gen_ai.client.token.usage histograms, plus the streaming timing histograms gen_ai.client.operation.time_to_first_chunk and gen_ai.client.operation.time_per_output_chunk, recorded for streamed inference calls

Usage

See the module docstring in observeinc.util.genai.handler for usage examples, including context manager and manual lifecycle patterns.

Context Management and Propagation

Invocation factory methods on TelemetryHandler (such as inference, workflow, tool, embedding, retrieval, and invoke_local_agent) accept an optional context keyword argument to manage context:

• context: An explicit OpenTelemetry Context to parent the span. When omitted, the current ambient context is used.

Conversation ID on Every Span

This fork sets gen_ai.conversation.id on every span, including tool, embeddings, retrieval and fetch-response spans, where semconv omits it. The value is the invocation’s conversation_id, which defaults to the one on the OTel Context. Setting conversation_id after creation also reaches the operations started under the invocation afterwards: those parented on its context, and, while nothing else has been attached on top of it, those using the current context. Set it in the thread or task that created the invocation, as with stop(). A value an instrumentation sets on attributes takes precedence.

Agent Name on Child Operations

This fork adds gen_ai.agent.name to the inference, embeddings, retrieval and fetch-response spans an agent runs, and to the metrics they record, which semconv does not define. It travels on the OTel Context the way the conversation id does: handler.inference(), embedding(), retrieval(), fetch_response() and tool() take an agent_name, and default to the enclosing agent’s, so the name reaches operations another instrumentation package creates.

As with the conversation id, an explicit agent_name also reaches the operations under that invocation. An agent always sets its own name, so the nearest agent wins, and one with no name hides an outer agent’s. A value an instrumentation sets itself, on attributes or metric_attributes, takes precedence.

User ID

This fork records the standard OpenTelemetry user.id attribute on every GenAI span started while a user id is set. Wrap the code that serves a user in using_user_id:

from observeinc.util.genai.user_id import using_user_id

with using_user_id("user-123"):
    agent.run(...)

The id travels on the OTel Context, so it reaches spans created by any instrumentation package, and by TelemetryHandler directly. It is not put in baggage, because baggage is sent in the headers of outgoing requests, including those to the model provider. To set it for a single operation, and the operations it starts, pass context=with_user_id("user-123") to a TelemetryHandler factory such as handler.inference(...). A value an instrumentation sets on attributes takes precedence.

Some libraries accept a user id of their own, such as a LangChain run’s user_id metadata. Set OBSERVE_GENAI_CAPTURE_USER_ID_FROM_LIBRARY to true to let instrumentations record it. It is off by default, so an id passed to a library does not start showing up in telemetry on upgrade. When it is on, the library’s id takes precedence over one set with using_user_id. The flag is read each time an instrumentation starts an operation.

Inference Token Metrics

This fork also records the token metrics semconv introduced in semantic-conventions-genai#374, on inference and embeddings operations. gen_ai.client.token.usage is still recorded alongside them.

• gen_ai.client.inference.usage.input_tokens, .output_tokens, .cache_read.input_tokens, .cache_write.input_tokens and .reasoning.output_tokens: counters, for totals and cost. Each carries gen_ai.token.modality, taken from the invocation’s per-modality token counts; the part no modality accounts for is unknown.

• gen_ai.client.inference.operation.input_tokens and .output_tokens: histograms, one measurement per operation.

Cache and reasoning tokens are already included in the input and output totals, so don’t add those counters to the input or output ones.

from opentelemetry import context
from observeinc.util.genai.invocation import (
    CLIENT_INFERENCE_CONTEXT_KEY,
    InferenceData,
)

data = InferenceData()
token = context.attach(context.set_value(CLIENT_INFERENCE_CONTEXT_KEY, data))
try:
    # Inner instrumented call detects data on context, suppresses its span,
    # and populates attributes onto data
    execute_operation()
finally:
    context.detach(token)

# Attributes captured by inner calls are available on data
print(data.response_model, data.usage_input_tokens)

from opentelemetry import context
from observeinc.util.genai.invocation import (
    CLIENT_INFERENCE_CONTEXT_KEY,
    InferenceData,
)

data = context.get_value(CLIENT_INFERENCE_CONTEXT_KEY)
if isinstance(data, InferenceData):
    # Outer operation is active: suppress span/metrics and publish into shared dataclass
    data.response_model = "gpt-4o-mini"
    data.usage_input_tokens = 10
else:
    # Emit telemetry normally
    ...

Stream execution context

SyncStreamWrapper and AsyncStreamWrapper scope each stream read and cleanup operation with _execution_context(). Context is restored before returning a chunk to the consumer. Generator send/throw and asend/athrow also use this scope when the underlying stream supports them.

For invocation-backed streams, pass the invocation to the base constructor (super().__init__(stream, invocation)) and call invocation.suspend() before returning the stream. The base wrapper then makes the suspended invocation current during reads, cleanup and finalization of an abandoned stream, so no _execution_context() override is needed. Keep finalization in the existing _on_stream_end and _on_stream_error hooks.

Modalities

observeinc.util.genai.types.Modality provides string enum members TEXT, IMAGE, VIDEO, AUDIO, and DOCUMENT. Use these constants when constructing BlobPart, FilePart, and UriPart or passing modality/token-count pairs to the inference invocation’s token setters:

from observeinc.util.genai.types import Modality, UriPart

image = UriPart(
    mime_type="image/png",
    modality=Modality.IMAGE,
    uri="https://example.com/image.png",
)

Members serialize and format as their string values, such as "image". Message parts continue to accept plain strings, including provider-specific modalities. Token setters continue to record only text, image, and audio; other modalities are ignored.

Modality replaces the previous Literal type alias. Annotations that also accept raw strings should use Modality | str rather than Modality alone. Custom modalities remain strings, not additional enum members.

Retrieval Documents

Set RetrievalInvocation.documents using observeinc.util.genai.types.RetrievalDocument objects:

from observeinc.util.genai.types import RetrievalDocument

with handler.retrieval(data_source_id="my-index") as invocation:
    invocation.documents = [RetrievalDocument(id="doc-1", score=0.9)]

The model contains only the optional id and score fields; unset fields serialize as JSON null. Documents are recorded in gen_ai.retrieval.documents only in SPAN_ONLY or SPAN_AND_EVENT content-capture mode. Passing dictionaries is deprecated, but existing dictionary payloads continue to serialize unchanged.

Environment Variables

This package relies on environment variables to configure capturing of message content. By default, message content will not be captured. Set the environment variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to one of:

• NO_CONTENT: Do not capture message content (default).

• SPAN_ONLY: Capture message content in spans only.

• EVENT_ONLY: Capture message content in events only.

• SPAN_AND_EVENT: Capture message content in both spans and events.

• ALLOW_LIST: Do not capture message content, and keep only allowlisted attributes. Specific to this fork; see Allowlist Mode.

An error message can quote the prompt, so under NO_CONTENT and EVENT_ONLY, which put no message content on spans, a failed span keeps its ERROR status and error.type but no status description. The gen_ai.client.inference.operation.details event carries error.type and never the message, in every mode. Specific to this fork.

When a LoggerProvider is configured, the package emits gen_ai.client.inference.operation.details events (with DEBUG severity) regardless of the content capturing mode. Message content is included (as dictionaries instead of JSON strings) only in EVENT_ONLY or SPAN_AND_EVENT mode. To suppress these events while keeping a LoggerProvider active, configure your logger or LogRecordProcessor to filter them out (see examples/suppress-inference-event).

Span Attributes

This package sets the following span attributes on LLM invocations:

Common attributes:

• gen_ai.operation.name: Str(chat)

• gen_ai.provider.name: Str(openai)

• gen_ai.request.model: Str(gpt-4o)

• server.address: Str(api.openai.com)

• server.port: Int(443)

Response attributes:

• gen_ai.response.finish_reasons: Slice([“stop”])

• gen_ai.response.model: Str(gpt-4o-2024-05-13)

• gen_ai.response.id: Str(chatcmpl-Bz8yrvPnydD9pObv625n2CGBPHS13)

• gen_ai.usage.input_tokens: Int(24)

• gen_ai.usage.output_tokens: Int(7)

• gen_ai.usage.cache_write.input_tokens: Int(10)

• gen_ai.usage.cache_read.input_tokens: Int(5)

Request parameter attributes (when provided):

• gen_ai.request.temperature: Float(0.7)

• gen_ai.request.top_p: Float(1.0)

• gen_ai.request.frequency_penalty: Float(0.0)

• gen_ai.request.presence_penalty: Float(0.0)

• gen_ai.request.max_tokens: Int(1024)

• gen_ai.request.stop_sequences: Slice([”\n”])

• gen_ai.request.seed: Int(42)

Content attributes (sensitive, requires content capturing enabled):

• gen_ai.input.messages: Str(‘[{“role”: “user”, “parts”: [{“content”: “hello world”, “type”: “text”}]}]’)

• gen_ai.output.messages: Str(‘[{“role”: “assistant”, “parts”: [{“content”: “hello back”, “type”: “text”}], “finish_reason”: “stop”}]’)

• gen_ai.system_instructions: Str(‘[{“content”: “You are a helpful assistant.”, “type”: “text”}]’)

Error attributes:

• error.type: Str(TimeoutError)

Embedding Span Attributes

This package also supports embedding invocation spans via the embedding context manager. For embedding invocations, the following attributes are set:

Common attributes:

• gen_ai.operation.name: Str(embeddings)

• gen_ai.provider.name: Str(openai)

• server.address: Str(api.openai.com)

• server.port: Int(443)

Request attributes:

• gen_ai.request.model: Str(text-embedding-3-small)

• gen_ai.embeddings.dimension.count: Int(1536)

• gen_ai.request.encoding_formats: Slice([“float”])

Response attributes:

• gen_ai.response.model: Str(text-embedding-3-small)

• gen_ai.usage.input_tokens: Int(24)

Capturing request iterators

Use IteratorCapture.from_property() for a named request property. It replaces that property’s value in kwargs and records values as the SDK reads them. Disabled captures and missing properties leave kwargs untouched and return None from snapshot().

from observeinc.util.genai.iterator_capture import IteratorCapture

messages_capture = IteratorCapture.from_property(
    kwargs, "messages", enabled=capture_content
)
try:
    response = sdk_call(**kwargs)
finally:
    messages = messages_capture.extract(get_input_messages)

For a bare value, pass capture.wrapped to the SDK:

capture = IteratorCapture(messages, enabled=capture_content)
try:
    response = sdk_call(messages=capture.wrapped)
finally:
    messages = capture.extract(get_input_messages)

extract(extractor) calls the instrumentation’s extractor with a snapshot, using () for unread iterators and None for disabled or missing properties. The extractor must accept that shape and return the telemetry content. extract() logs snapshot or conversion errors and returns None. Use snapshot() directly when no conversion is needed.

Nested built-in dictionaries, lists, and tuples are supported. File objects pass through unchanged. Snapshots contain only the consumed portion of each iterator; async iterators and custom containers are not captured.

Installation

pip install observe-util-genai

For upload support (requires fsspec):

pip install "observe-util-genai[upload]"

Design Document

The design document for the OpenTelemetry GenAI Utils can be found at: Design Document

References

• OpenTelemetry Project

• OpenTelemetry GenAI semantic conventions