AgentWatch — Python SDK

Predict when production AI agents are degrading — before they visibly fail.

AgentWatch watches your agent's traces and runs statistical change-point detection (CUSUM/EWMA), an LLM-judge correctness gate, and semantic-drift scoring server-side to catch quality collapse before your error rate moves. This SDK just captures traces and ships them — it has zero required dependencies and never adds latency to or raises errors in your agent's request path.

Install

pip install agentwatch-dev
# optional, only for instrument_openai():
pip install "agentwatch-dev[openai]"

The distribution is named agentwatch-dev (after agentwatch.dev); the import name is just agentwatch.

Configure

import agentwatch
agentwatch.configure(api_key="aw_live_...", agent_id="<agent-uuid>")

Self-hosting? Pass endpoint="https://your-host" to configure(...).

Option A — auto-instrument (one line)

Works for any OpenAI-compatible client — OpenAI, Groq, Cerebras, Together, vLLM — because they all use the openai client with a different base_url.

from openai import OpenAI

client = OpenAI(base_url="https://api.groq.com/openai/v1", api_key=GROQ_API_KEY)
agentwatch.instrument_openai(client)

# Every chat completion from here on is traced automatically:
client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Refund order #47829"}],
)

Option B — wrap your agent function

@agentwatch.watch()
def run_agent(query: str) -> str:
    ...
    return answer

# async is supported too:
@agentwatch.watch()
async def run_agent_async(query: str): ...

watch() captures the first argument as the input and the return value as the output. If the return value is an OpenAI-shaped response, it also extracts the model, token counts, and tool names automatically.

Option C — manual trace

client = agentwatch.AgentWatchClient(api_key="aw_live_...", agent_id="<uuid>")
client.trace(
    input="Refund order #47829",
    output="I've issued the refund…",
    model="llama-3.3-70b-versatile",
    latency_ms=812,
    prompt_tokens=220, completion_tokens=180, total_tokens=400,
    tools_used=["search_kb", "issue_refund"],
)

Notes

• Fire-and-forget: traces are POSTed on a daemon thread; delivery failures are swallowed so monitoring can never break your agent.
• Metadata: attach tags via configure(..., metadata={"env": "prod"}) or per-call @agentwatch.watch(metadata={"version": "2.1"}).
• Streaming (stream=True) responses aren't fully captured yet — the trace is still recorded, but token/output extraction may be partial.

MIT licensed.