evalwatch

Python client for sending LLM traces to EvalWatch — a SaaS observability and evaluation platform for LLM products.

Install

# Sync client only (uses requests)
pip install evalwatch

# Sync + async client (adds httpx)
pip install 'evalwatch[async]'

Quick start — sync

Get an API key from your EvalWatch dashboard (/settings), then instrument your LLM calls:

from evalwatch import EvalWatchClient

client = EvalWatchClient("https://your-evalwatch-instance.com", api_key="sk-...")

# 1. Manual
client.send_trace(
    prompt="What is the capital of France?",
    response="Paris.",
    latency_ms=120,
    model_name="gpt-4o",
    provider="openai",
    token_usage=30,
)

# 2. Context manager — measures latency automatically
with client.capture(prompt="...", model_name="gpt-4o", provider="openai") as trace:
    response = openai_client.chat.completions.create(...)
    trace.finish(response=response.choices[0].message.content, token_usage=response.usage.total_tokens)

# 3. Decorator — wraps any function whose first arg is the prompt
@client.watch(model_name="gpt-4o", provider="openai")
def call_llm(prompt: str) -> str:
    return openai_client.complete(prompt)

# Retry on transient failures (ConnectionError or 5xx)
client.send_trace(..., max_retries=2)

Quick start — async

from evalwatch import AsyncEvalWatchClient

client = AsyncEvalWatchClient("https://your-evalwatch-instance.com", api_key="sk-...")

# 1. Manual (awaitable)
await client.send_trace(
    prompt="What is the capital of France?",
    response="Paris.",
    latency_ms=120,
    model_name="gpt-4o",
    provider="openai",
    token_usage=30,
)

# 2. Async context manager
async with client.capture(prompt="...", model_name="gpt-4o") as trace:
    response = await openai_client.chat.completions.acreate(...)
    await trace.finish(response=response.choices[0].message.content)

# 3. Async decorator
@client.watch(model_name="gpt-4o", provider="openai")
async def call_llm(prompt: str) -> str:
    return await openai_client.acomplete(prompt)

All methods send traces to POST /api/v1/observe/traces and are visible in the Traces dashboard immediately.

License

MIT