telemetry-dev-litellm

LiteLLM integration for the telemetry.dev Python SDK. It instruments LiteLLM completion, Responses, embedding, and rerank APIs, including their async and streaming variants, and litellm.Router methods when available.

Install

pip install telemetry-dev telemetry-dev-litellm
# or
uv add telemetry-dev telemetry-dev-litellm

Requires Python >=3.10,<3.14.

Initialize telemetry.dev

import telemetry_dev
from telemetry_dev_litellm import instrument_litellm

telemetry_dev.init()  # reads TELEMETRY_DEV_API_KEY and OTLP settings
instrument_litellm()

The core SDK fails open: without an API key, LiteLLM calls still run and telemetry is a no-op.

Per-call drop-ins

Use the exported functions as drop-in replacements when you do not want global monkey-patching:

from telemetry_dev_litellm import completion

response = completion(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Say hi"}],
)

Available drop-ins: completion, acompletion, responses, aresponses, embedding, aembedding, rerank, arerank. Each resolves the current litellm.<function> at call time. If global instrumentation is already active, it delegates to the globally wrapped function so the call produces exactly one span.

Global instrumentation

import litellm
from telemetry_dev_litellm import instrument_litellm, uninstrument_litellm

instrument_litellm()
response = litellm.completion(model="gpt-4o-mini", messages=[...])
uninstrument_litellm()

instrument_litellm() patches these package attributes when LiteLLM provides them:

• litellm.completion
• litellm.acompletion
• litellm.embedding
• litellm.aembedding
• litellm.rerank
• litellm.arerank
• litellm.responses
• litellm.aresponses

It is thread-safe and idempotent. uninstrument_litellm() restores the original objects and is also idempotent.

Import-order limitation: references captured before instrumentation, such as from litellm import completion, keep pointing at the original function and are not traced. Instrument at process startup or use the telemetry_dev_litellm drop-ins.

Router wrapping

Global instrumentation traces Router deployment attempts because LiteLLM Router dispatches through the patched package attributes at call time:

import litellm
from telemetry_dev_litellm import instrument_litellm

instrument_litellm()
router = litellm.Router(model_list=[...])
router.completion(model="my-group", messages=[...])  # one span per deployment attempt

Use wrap_router(router) for an enclosing Router-level span:

import litellm
from telemetry_dev_litellm import instrument_litellm, wrap_router

instrument_litellm()
router = wrap_router(litellm.Router(model_list=[...]))
router.completion(model="my-group", messages=[...])

With both wrap_router() and instrument_litellm() active, the Router span is the parent and each deployment attempt is a child span. Router-injected LiteLLM metadata such as model_group and deployment is forwarded as td.metadata.* on inner attempt spans when LiteLLM provides it.

Internal LiteLLM num_retries retries happen inside a single wrapped call, so they are represented as one span whose duration covers all attempts. Router retries and fallbacks are separate deployment calls and can produce separate child spans under global instrumentation.

Captured fields

Chat spans use type="generation" and emit gen_ai.operation.name="chat":

• request model, provider, messages, sampling parameters (including top_k), stop sequences, seed, frequency and presence penalties
• response_format as gen_ai.output.type (json for json_object/json_schema/pydantic models)
• response model, response id, output messages, finish reasons, usage, cost when available
• metadata={...} passed to LiteLLM as td.metadata.*
• errors via the core SDK's error.type and exception event mapping

Embedding spans use type="embedding" and emit gen_ai.operation.name="embeddings":

• request model, provider, input, metadata
• response model, usage, cost when available
• no embedding vectors as output

Rerank spans emit gen_ai.operation.name="rerank":

• request model, provider, query, documents, ranking options, and metadata
• response id, ranked results and scores, result count, provider search units, token usage, and cost when available

Responses spans emit gen_ai.operation.name="chat" and capture request input, instructions, sampling options, metadata, response output/status/id, usage, cost, and provider. Streaming Responses events are handled separately from chat chunks; terminal failure and error events mark the span as failed, and early close records the partial output collected so far.

Streamed chat chunks and Responses reconstruction are retained up to 48 KiB and 1,000 items. Non-streamed Responses input and output and rerank input and results are retained up to 64 KiB and 1,024 items. The configured max_attribute_length does not shrink these retention bounds; the core applies it to the exported attribute after the mask runs. A chat stream that is cut at its bound, closed early, or fails, and a Responses stream that is cut at its bound, ends without a terminal response, or ends with an error event set telemetry.dev.capture.truncated. With a mask configured, the incomplete reconstructed output is omitted because the mask cannot inspect the complete value.

Aggregate usage keys are input_tokens, output_tokens, total_tokens, cache_read_input_tokens, cache_creation_input_tokens, and reasoning_output_tokens. The core contract also includes modality fields; Responses usage emits text, image, and audio input/output token details when LiteLLM exposes them.

Streaming

stream=True chat calls return a proxy around LiteLLM's CustomStreamWrapper that supports both sync and async iteration, context managers, close(), aclose(), and attribute delegation to the wrapped stream.

The integration never mutates request arguments. It does not inject stream_options.include_usage. On stream completion, early close, or mid-stream error, it rebuilds the best available response with litellm.stream_chunk_builder(...) and ends the span once. time_to_first_chunk_ms is recorded on the first chunk. Stream usage is best-effort: LiteLLM aggregates usage chunks when the provider sends them and otherwise may estimate counts from the chunks and request messages. Some LiteLLM versions instead expose zero usage for streams without provider usage, as observed with 1.104.2. When a stream produced output and its usage is zero input and zero output tokens with no other positive token count, the integration records no usage or cost instead of zero. To get the provider's own counts, pass stream_options={"include_usage": True} to providers that support it.

Provider attribution

LiteLLM provider names are mapped to OpenTelemetry well-known gen_ai.provider.name values where one exists:

Unlisted providers pass through unchanged, for example ollama, openrouter, together_ai, and fireworks_ai.

Cost

LiteLLM's synchronous response metadata is used when available: _hidden_params.response_cost maps to gen_ai.usage.cost. If that field is absent, the integration tries litellm.completion_cost(completion_response=...) and silently omits cost if LiteLLM cannot price the model.

Limitations

• Instrumented surfaces are limited to completion, acompletion, responses, aresponses, embedding, aembedding, rerank, arerank, and the corresponding methods available on Router via wrap_router. text_completion, image/audio/batch APIs, video APIs such as video_generation and video_status, and provider-specific APIs are out of scope for this package version.
• References imported from LiteLLM before instrument_litellm() are not patched.
• Internal num_retries retries are not separate spans; Router deployment attempts are.
• OTel's built-in-tools scenario is not portable across LiteLLM providers and is not covered by the examples.
• Multimodal output is not covered; LiteLLM chat returns text/tool calls, while image and audio generation use APIs outside this integration.

Development

uv sync
uv run pytest
uv run ruff format .
uv run ruff check .
uv run pyright

Repository-level commands run this package together with the core Python SDK:

pnpm run py:sync
pnpm run py:check
pnpm run py:test