Monte Carlo OpenTelemetry SDK

This library provides a Python SDK for tracing applications with OpenTelemetry for use with Monte Carlo's AI Observability solution.

To evaluate the effectiveness of AI agents, the first step is capturing the prompts sent to an LLM and the completions returned. The next challenge is categorizing these LLM calls, since different types of LLM calls require different evaluation approaches.

This SDK not only streamlines OpenTelemetry tracing setup, but it also makes it easy to add custom attributes to spans, enabling you to filter and select different subsets of spans for evaluation.

This is alpha software. The API is subject to change.

Installation

Install the SDK

Requires Python 3.10 or later.

$ pip install montecarlo-opentelemetry

Install the instrumentation package(s) for the AI libraries you want to trace.

The Monte Carlo SDK can work with existing instrumentation for AI libraries to capture traces automatically. Choose the instrumentation library that matches the library you are using.

# For Langchain/LangGraph
$ pip install "opentelemetry-instrumentation-langchain<=0.53.4"

# For OpenAI
$ pip install "opentelemetry-instrumentation-openai<=0.53.4"

See a selection of available instrumentation libraries below.

Quick Start

Set up Tracing in Your Application

# Import the Monte Carlo SDK.
import montecarlo_opentelemetry as mc

# Import the AI client library (Anthropic in this example.)
from anthropic import Anthropic

# Import the corresponding instrumentation library.
from opentelemetry.instrumentation.anthropic import AnthropicInstrumentor

# Create an Instrumentor object
anthropic_instrumentor = AnthropicInstrumentor()

# Set up tracing.
mc.setup(
    agent_name="my-agent",
    otlp_endpoint="http://localhost:4318/v1/traces",
    instrumentors=[anthropic_instrumentor],
)

# Use decorator to add a Monte Carlo workflow attribute.
@mc.trace_with_workflow("parent-function", "my-workflow")
def parent():
    child()

# Use decorator to add a Monte Carlo task attribute.
@mc.trace_with_task("child-function", "my-task")
def child():
    message = Anthropic().messages.create(
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": "Hello world!",
            }
        ],
        model="claude-sonnet-4-20250514",
    )

Add MonteCarlo API credentials as environment variables

If you are sending OpenTelemetry traces to a Monte Carlo OTLP Ingestion endpoint, you will need to add your Monte Carlo API credentials as environment variables when running your application. See our docs on generating API keys.

There are two different ways to add your Monte Carlo API credentials as environment variables. Choose the option that best fits your needs.

1. Use Monte Carlo environment variables. Similar to how environment variables are used for the Pycarlo SDK, you can set separate environment variables for your Monte Carlo API ID and API token.

   $ export MCD_DEFAULT_API_ID=<your-api-id>
   $ export MCD_DEFAULT_API_TOKEN=<your-api-token>
   

2. Use the standard OpenTelemetry environment variable for headers. You can add both your Monte Carlo API ID and API token to the OTEL_EXPORTER_OTLP_HEADERS environment variable. The value of this environment variable should be a comma-separated list of key=value pairs, where the keys are: x-mcd-id and x-mcd-token.

   $ export OTEL_EXPORTER_OTLP_HEADERS="x-mcd-id=<your-api-id>,x-mcd-token=<your-api-token>"
   

Configuring the span processor

By default, mc.setup() configures a BatchSpanProcessor that buffers spans and flushes them on a timer. This is the right choice for long-running services because batching reduces export overhead.

The default exporter pipeline is tuned for the large spans that AI/LLM instrumentation produces (prompt and completion content can make a single span several MB):

• gzip compression is on by default, shrinking OTLP request bodies over the wire.
• max_export_batch_size defaults to 128 (smaller than OpenTelemetry's default of 512), keeping each OTLP request small enough to stay under the collector's body-size limit.

Both are lossless. If you supply your own span_processor (below), these defaults do not apply — configure compression and batch size on your processor as needed.

If you need different export behavior, pass your own processor with the span_processor kwarg:

mc.setup(
    agent_name="my-agent",
    otlp_endpoint="http://localhost:4318/v1/traces",  # required by signature; ignored when span_processor is set
    instrumentors=[LangchainInstrumentor()],          # Match the instrumentor to the AI library/framework you are using.
    span_processor=my_custom_processor,
)

The processor you pass replaces only the default exporter pipeline. mc.MonteCarloSpanProcessor — which propagates montecarlo.* attributes through the trace — is always installed on the tracer provider mc.setup() creates. If another library has already set the global tracer provider, that provider is used instead, and neither Monte Carlo's processor nor the one you pass is installed on it; see When another library already set the TracerProvider.

Serverless and other suspendable runtimes

In runtimes that can freeze the Python process between invocations — AWS Lambda, Google Cloud Run services, and similar — BatchSpanProcessor's timer-driven flush will not run reliably. Spans queued in the buffer are lost when the process is suspended and the runtime moves on without resuming it.

In these environments, use SimpleSpanProcessor, which exports each span synchronously when it ends:

import os

import montecarlo_opentelemetry as mc
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.langchain import LangchainInstrumentor
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

mcd_headers = {
    "x-mcd-id": os.environ["MCD_DEFAULT_API_ID"],
    "x-mcd-token": os.environ["MCD_DEFAULT_API_TOKEN"],
}

# SimpleSpanProcessor flushes each span before the runtime can suspend the
# process. BatchSpanProcessor would queue spans and lose them at freeze.
exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces", headers=mcd_headers)
span_processor = SimpleSpanProcessor(exporter)

mc.setup(
    agent_name="my-agent",
    otlp_endpoint="http://localhost:4318/v1/traces",  # required by signature; ignored when span_processor is set
    instrumentors=[LangchainInstrumentor()],          # Match the instrumentor to the AI library/framework you are using.
    span_processor=span_processor,
)

Auth headers note: When you supply a custom span_processor, mc.setup() does not inject the MCD_DEFAULT_API_ID / MCD_DEFAULT_API_TOKEN headers — that injection only happens when mc.setup() builds the default exporter. The example above passes those headers explicitly. If you use OTEL_EXPORTER_OTLP_HEADERS instead, OTLPSpanExporter reads it automatically and you can omit the headers= argument.

The trade-off is per-span latency: each span ending blocks on an OTLP round-trip. That cost is acceptable when the alternative is losing spans entirely.

When another library already set the TracerProvider

OpenTelemetry lets a process set its global tracer provider only once, and the first one set wins. Some frameworks install their own while they start up, before your agent code runs — Google ADK's adk web does this. When that happens, mc.setup() cannot install Monte Carlo's provider: the existing provider stays in place, mc.setup() logs a warning, and spans are handled by the provider that is already there.

Add Monte Carlo to that provider with the two functions:

import montecarlo_opentelemetry as mc

mc.add_processor()                                  # montecarlo.* attributes on spans
mc.add_exporter("http://localhost:4318/v1/traces")  # Monte Carlo's OTLP pipeline

Both default to the global provider, both accept an explicit provider= instead, and both are idempotent: adding the same processor, or the same endpoint, twice does nothing. That matters because a framework can import your module more than once in a single process (Google ADK's --reload_agents does exactly that). Call them before the spans you care about, since attributes are added when a span starts. If you build a provider yourself, you can also add the processor directly with provider.add_span_processor(mc.MonteCarloSpanProcessor()); add_processor() does the same, once per provider.

add_exporter() adds Monte Carlo's tuned pipeline — gzip, max_export_batch_size=128, and credentials from MCD_DEFAULT_API_ID / MCD_DEFAULT_API_TOKEN, or from OTEL_EXPORTER_OTLP_TRACES_HEADERS or OTEL_EXPORTER_OTLP_HEADERS when those are not set. A different endpoint is added as well, so Monte Carlo can be one of several destinations. Nothing is added automatically: if you don't call it and the provider in place doesn't export to Monte Carlo, no traces arrive.

Two things belong to the provider itself and cannot be changed once it exists: service.name and the span attribute limit (mc.setup() uses 1024). The OpenTelemetry SDK reads OTEL_SERVICE_NAME and OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT when the provider is created, so set them in the environment that launches the process. A .env file loaded after startup is too late — add_processor() warns when OTEL_SERVICE_NAME arrived after the provider was created.

To keep a custom pipeline — SimpleSpanProcessor in a suspendable runtime, for example — add your own exporter instead, with the same settings mc.setup() uses:

import os

import montecarlo_opentelemetry as mc
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http import Compression
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

mc.add_processor()

exporter = OTLPSpanExporter(
    endpoint="http://localhost:4318/v1/traces",  # Your endpoint, including the /v1/traces path.
    headers={
        "x-mcd-id": os.environ["MCD_DEFAULT_API_ID"],
        "x-mcd-token": os.environ["MCD_DEFAULT_API_TOKEN"],
    },
    compression=Compression.Gzip,
)
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(exporter, max_export_batch_size=128)
)

Don't add this exporter when the provider already exports to Monte Carlo — every span would be sent twice. And you own the processor you create, including its flush behavior; see Serverless and other suspendable runtimes.

One provider, one service name, many agents

A framework that hosts several agents in one process has one tracer provider, and therefore one service.name. Label each agent per trace instead, with montecarlo.agent_name baggage, which Monte Carlo's span processor copies onto every span that starts while it is set:

from opentelemetry import baggage, context

token = context.attach(baggage.set_baggage("montecarlo.agent_name", "business-analyst-agent"))
try:
    run_the_agent()
finally:
    context.detach(token)

Set it per run, from whatever hook the framework gives you — an ADK plugin's before-run callback, for instance — rather than from a process-wide default, so concurrent agents don't label each other's spans.

Troubleshooting

• The agent shows up as unknown_service. The provider was created before OTEL_SERVICE_NAME was available, which is what a configuration file loaded after startup produces. Set the variable in the environment that launches the process, or label per trace with montecarlo.agent_name baggage as above.
• The ingestion endpoint rejects an export as an invalid key id. A credential is wrapped in quotes — x-mcd-id='abc123' — which shells and .env files often keep as part of the value. When mc.setup() or mc.add_exporter() builds the exporter, Monte Carlo logs a warning naming the header (never the value).
• Traces stop being labeled after editing agent code. Google ADK's --reload_agents re-imports agent modules in the same process, so module-level code runs again. Use add_processor() and add_exporter(), which are idempotent, rather than adding processors and exporters by hand.

Example

Example Application

To see how the Monte Carlo SDK can be used to add identifying attributes to spans, let's look at an example application that's slighly larger than the Quick Start one.

This is a simplified "Travel Assistant" agent that can make hotel and flight reservations.

# Fake LLM library that can make a call to an LLM.
from example.LLMLibrary import call_llm

def travel_assistant():
    flight_assistant()
    hotel_assistant()

def flight_assistant():
    plan_flight()
    book_flight()

def plan_flight():
    pass

def book_flight():
    call_llm()

def hotel_assistant():
    search_for_hotel()
    book_hotel()

def search_for_hotel():
    call_llm()

def book_hotel():
    call_llm()

If we traced each function in this application, the structure of the trace would look like this:

travel_assistant
├── flight_assistant
│   ├── plan_flight
│   └── book_flight
│       └── call_llm
└── hotel_assistant
    ├── search_for_hotel
    │   └── call_llm
    └── book_hotel
        └── call_llm

In order to differentiate between different types of LLM calls, it would be helpful to add identifying attributes to spans. That way we could tell if an LLM call was part of a workflow managed by the flight assistant, or if it was part of the hotel booking task in the workflow managed by the hotel assistant.

Adding Attributes with the Monte Carlo SDK

Let's see how we can use the Monte Carlo SDK to enhance the tracing data with identifying attributes.

import montecarlo_opentelemetry as mc

# Fake LLM library that can make a call to an LLM.
from example.LLMLibrary import call_llm

# Fake LLM library instrumentation that will automatically create spans
# each time call_llm() is called.
from example.LLMLibrary.instrumentation import LLMInstrumentor

mc.setup(
    agent_name="travel-assistant",
    otlp_endpoint="http://localhost:4318/v1/traces",
    instrumentors=[LLMInstrumentor()],
)

@mc.trace_with_tags(span_name="travel_assistant", tags=["travel", "v1"])
def travel_assistant():
    flight_assistant()
    hotel_assistant()

@mc.trace_with_workflow(span_name="flight_assistant", workflow_name="flight")
def flight_assistant():
    plan_flight()
    book_flight()

@mc.trace_with_task(span_name="plan_flight", task_name="plan")
def plan_flight():
    pass

@mc.trace_with_task(span_name="book_flight", task_name="book")
def book_flight():
    call_llm()

@mc.trace_with_workflow(span_name="hotel_assistant", workflow_name="hotel")
def hotel_assistant():
    search_for_hotel()
    book_hotel()

@mc.trace_with_task(span_name="search_for_hotel", task_name="search")
def search_for_hotel():
    call_llm()

# Arguments can also be passed positionally.
@mc.trace_with_task("book_hotel", "book")
def book_hotel():
    call_llm()

Because montecarlo.* attributes propagate from parent to child spans, the call_llm spans will contain all of the montecarlo.* attributes were added to spans that occur above it in the trace hierarchy.

For example, the call_llm span for book_flight will not only have the montecarlo.task = "book" attribute that we added directly, but also the montecarlo.workflow = "flight" added on the flight_assistant span, and the montecarlo.tags = "travel,v1" attributes added on the travel_assistant span.

That results in the following trace structure:

travel_assistant            <-- montecarlo.tags = "travel,v1"
│
├── flight_assistant        <-- montecarlo.workflow = "flight"
|   |                       <-- montecarlo.tags = "travel,v1"
|   |
│   ├── plan_flight         <-- montecarlo.task = "plan"
|   |                       <-- montecarlo.workflow = "flight"
|   |                       <-- montecarlo.tags = "travel,v1"
|   |
│   └── book_flight         <-- montecarlo.task = "book"
│       |                   <-- montecarlo.workflow = "flight"
│       |                   <-- montecarlo.tags = "travel,v1"
│       |
│       └── call_llm        <-- montecarlo.task = "book"
│                           <-- montecarlo.workflow = "flight"
│                           <-- montecarlo.tags = "travel,v1"
│
└── hotel_assistant         <-- montecarlo.workflow = "hotel"
    |                       <-- montecarlo.tags = "travel,v1"
    |
    ├── search_for_hotel    <-- montecarlo.task = "search"
    │   |                   <-- montecarlo.workflow = "hotel"
    │   |                   <-- montecarlo.tags = "travel,v1"
    │   |
    │   └── call_llm        <-- montecarlo.task = "search"
    │                       <-- montecarlo.workflow = "hotel"
    │                       <-- montecarlo.tags = "travel,v1"
    │
    └── book_hotel          <-- montecarlo.task = "book"
        |                   <-- montecarlo.workflow = "hotel"
        |                   <-- montecarlo.tags = "travel,v1"
        |
        └── call_llm        <-- montecarlo.task = "book"
                            <-- montecarlo.workflow = "hotel"
                            <-- montecarlo.tags = "travel,v1"

Capturing Span I/O

The trace_with_attributes, trace_with_task, and trace_with_workflow decorators accept an optional capture_io=True parameter (default False) that records the decorated function's inputs and return value directly into span attributes. This data is used by offline replay and evaluation tooling.

@mc.trace_with_task("analyze-monitor", "analyze", capture_io=True)
def analyze(monitor_uuid: str, threshold: int = 10):
    ...

What gets recorded

When capture_io=True is set, the decorator serializes the call's arguments — including positional arguments mapped to their parameter names — and writes them to the span attribute montecarlo.span_input. The return value is written to montecarlo.span_output.

The attributes use a structured envelope:

• montecarlo.span_input — {"inputs": {"monitor_uuid": "abc-123", "threshold": 10}, "tags": [], "metadata": {}, "kwargs": {}}
• montecarlo.span_output — {"outputs": <return value>, "kwargs": {}}

Values that aren't JSON-serializable are recorded as a <unserializable …> placeholder rather than their repr. Note that nested dataclass and Pydantic model fields are expanded — a config object with a nested credential field would be captured in full.

Why it's off by default

capture_io is opt-in per decoration so you control both data volume and which calls expose their arguments. Large inputs or outputs (long documents, big lists) can increase span size significantly, and not every call is appropriate to record — see the warning below.

Warning: With capture_io=True, call inputs and return values are serialized verbatim into span attributes and exported to your configured telemetry backend, where they are searchable, retained for the backend's retention window, often under broader access than your secret store, and not rotated when a secret rotates. Do not enable capture_io on calls that carry secrets, credentials, tokens, or PII. Values that aren't JSON-serializable are recorded as a <unserializable …> placeholder (not their repr), but nested dataclass/Pydantic fields are expanded — so a config object with a nested credential field would be captured verbatim.

For LLM prompt content specifically, see the Tracing LLM Calls Manually section below, which covers how to pass a redacted prompt list to create_llm_span while sending the un-redacted prompts to the model.

Tracing LLM Calls Manually

Typically, an instrumentation library will be used to automatically trace LLM calls. When that's not possible, the create_llm_span context manager can be used to create a span for the LLM call manually.

The create_llm_span context manager will set request-related attributes. Since provider, model, operation, and prompts are known before the LLM call is made, they should be passed to the context manager so that the appropriate span attributes can be added automatically. Response-related attributes need to be added with the helper functions after the LLM call.

It is possible to record a list of prompts as attributes that is different than the prompts sent to the LLM. If you have sensitive data that should not be recorded as span attributes, you can pass a modified list of prompts to create_llm_span, and then pass the un-redacted prompts to the LLM.

import montecarlo_opentelemetry as mc

# Fake LLM library that can make a call to an LLM.
from example.LLMLibrary import call_llm

prompts_to_record = [
    {"role": "system", "content": "You are a world-class greeter."},
    {"role": "user", "content": "Say hello to Bob."},
    {"role": "assistant", "content": "Hello Bob!"},
]

prompts_to_send = [
    {"role": "system", "content": "You are a world-class greeter."},
    {"role": "user", "content": "Say hello to Bob. Use SENSITIVE DATA."},
    {"role": "assistant", "content": "Hello Bob!"},
]

with mc.create_llm_span(
    span_name="example-span",
    provider="llm-provider",
    model="llm-model",
    operation="chat",
    prompts_to_record=prompts_to_record,
) as span:
    # Make LLM call.
    #
    # We are sending un-redacted prompts to the LLM. The LLM will see
    # "SENSITIVE DATA", but it won't be recorded as a span attribute.
    resp = call_llm(prompts_to_send)

    # Add response attributes to span.
    #
    # Assume that the response object has attributes like model, completions, etc.
    mc.add_llm_response_model(span, resp.model)
    mc.add_llm_completions(span, resp.completions)
    mc.add_llm_tokens(
        span,
        resp.prompt_tokens,
        resp.completion_tokens,
        resp.total_tokens,
        resp.cache_creation_input_tokens,
        resp.cache_read_input_tokens,
    )

License

Apache 2.0 - See the LICENSE for more information.

Security

See SECURITY.md for more information.

Available Instrumentation Packages

Note: Some packages have version constraints. See Version Compatibility below.

• opentelemetry-instrumentation-agno
• opentelemetry-instrumentation-alephalpha
• opentelemetry-instrumentation-anthropic
• opentelemetry-instrumentation-bedrock
• opentelemetry-instrumentation-chromadb
• opentelemetry-instrumentation-cohere
• opentelemetry-instrumentation-crewai
• opentelemetry-instrumentation-google-generativeai
• opentelemetry-instrumentation-groq
• opentelemetry-instrumentation-haystack
• opentelemetry-instrumentation-lancedb
• opentelemetry-instrumentation-langchain
• opentelemetry-instrumentation-llamaindex
• opentelemetry-instrumentation-marqo
• opentelemetry-instrumentation-mcp
• opentelemetry-instrumentation-milvus
• opentelemetry-instrumentation-mistralai
• opentelemetry-instrumentation-ollama
• opentelemetry-instrumentation-openai
• opentelemetry-instrumentation-openai-agents
• opentelemetry-instrumentation-pinecone
• opentelemetry-instrumentation-qdrant
• opentelemetry-instrumentation-replicate
• opentelemetry-instrumentation-sagemaker
• opentelemetry-instrumentation-together
• opentelemetry-instrumentation-transformers
• opentelemetry-instrumentation-vertexai
• opentelemetry-instrumentation-voyageai
• opentelemetry-instrumentation-watsonx
• opentelemetry-instrumentation-weaviate
• opentelemetry-instrumentation-writer

Version Compatibility

Newer versions of some instrumentation packages have adopted a different attribute format that will be supported in a future release.

Known incompatible versions:

Package	Last compatible version	First incompatible version
opentelemetry-instrumentation-anthropic	0.53.4	0.54.0
opentelemetry-instrumentation-openai	0.53.4	0.55.0
opentelemetry-instrumentation-langchain	0.53.4	0.55.0
opentelemetry-instrumentation-crewai	0.55.4	0.56.0
opentelemetry-instrumentation-bedrock	0.56.1	0.57.0

For other instrumentation packages, version 0.57.0 or earlier is recommended as they may adopt the new format in future releases.