LLM Watch SDK

Lightweight SDK to send LLM usage telemetry to the LLM Watch backend.

Install

pip install llm-watch-sdk

Quickstart

LLM Watch reads config from env by default:

LLMWATCH_BACKEND_URL=http://127.0.0.1:8000
LLMWATCH_API_KEY=YOUR_PROJECT_KEY

from llm_watch import LLMWatch

watch = LLMWatch()  # reads env

You can still pass backend_url and project_api_key explicitly.

from openai import OpenAI

client = OpenAI(api_key="OPENAI_KEY")
llm = watch.instrument_openai(client, model="gpt-4o-mini")

# trace() is optional; it groups related calls
with watch.trace(actor_id="user-42"):
    resp = llm.invoke({
        "messages": [{"role": "user", "content": "hello"}],
    })

Providers

Instrument a provider client, then use the same .invoke(...) pattern across providers.

# OpenAI (recommended unified invoke)
from openai import OpenAI

client = OpenAI(api_key="...")
llm = watch.instrument_openai(client, model="gpt-4o-mini")

with watch.trace(actor_id="user-42"):
    resp = llm.invoke({
        "messages": [{"role": "user", "content": "hello"}],
    })

# Gemini (google.genai)
from google import genai

client = genai.Client(api_key="...")
llm = watch.instrument_gemini(client, model="models/gemini-pro-latest")

with watch.trace(actor_id="user-42"):
    resp = llm.invoke({"contents": "hello"})

# Bedrock (create or wrap a client)
llm = watch.instrument_bedrock(region="eu-north-1", model_id="arn:aws:bedrock:...")

with watch.trace(actor_id="user-42"):
    resp = llm.invoke({
        "messages": [{"role": "user", "content": [{"text": "hello"}]}],
        "inferenceConfig": {"maxTokens": 64},
    })

Optional drop-in wrappers

If you prefer to keep your existing call sites, you can use drop-in wrappers:

# OpenAI drop-in (keeps .chat.completions.create)
client = OpenAI(api_key="...")
client = watch.wrap_openai(client, model="gpt-4o-mini")
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "hello"}],
)

# Bedrock drop-in (wrap existing boto3 client)
import boto3
br = boto3.client("bedrock-runtime", region_name="eu-north-1")
br = watch.instrument_bedrock(client=br, model_id="arn:aws:bedrock:...")
br.invoke_model(
    modelId="arn:aws:bedrock:...",
    body=b'{"messages":[{"role":"user","content":[{"text":"hello"}]}]}',
    contentType="application/json",
    accept="application/json",
)

Manual logging (if you prefer)

watch.log_llm_call(
    provider="openai",
    model="gpt-4o-mini",
    input_tokens=120,
    output_tokens=320,
    total_tokens=440,
    latency_ms=780,
    actor_id="user-42",
)

watch.trace() is optional. If you don’t create a trace, LLM Watch auto-creates one per call. Use trace() to group calls under a single workflow.

Production checklist

• Store LLMWATCH_API_KEY in your secrets manager (do not hardcode).
• Set LLMWATCH_BACKEND_URL via environment (per‑environment endpoint).
• If you skip trace(), each call still logs with an auto trace/span.
• Add request timeouts and retry policies to your LLM client (SDK never blocks your app).
• Capture actor context (actor_id, actor_type, actor_name) for accountability.
• Use one key per project/workspace for clean separation of data.