LLM Observe SDK

Automatic cost tracking and observability for LLM applications

Track costs, tokens, latency, and errors for OpenAI, Anthropic, and other LLM providers with just 2 lines of code.

Features

• ✅ Auto-instrumentation - No code changes needed beyond initialization
• 📊 Real-time tracking - Costs, tokens, latency, errors
• 🔍 Hierarchical tracing - Track agents, tools, and workflows
• 👥 Customer attribution - Track costs per end-user
• 🎯 Zero overhead - Async buffering, sub-millisecond impact
• 🔒 Privacy-first - Optional prompt/response capture
• 🚀 Production-ready - Battle-tested, scalable architecture

Supported Providers

• OpenAI (GPT-4, GPT-3.5, embeddings, images, audio, etc.)
• Pinecone (vector database operations)
• More coming soon! (Anthropic, Cohere, etc.)

Quick Start

1. Install

pip install llmobserve

2. Get Your API Key

Sign up at llmobserve.com to get your API key.

3. Add 2 Lines of Code

from llmobserve import observe
from openai import OpenAI

# Initialize observability
observe(
    collector_url="https://api.llmobserve.com",
    api_key="llmo_sk_your_key_here"
)

# Use OpenAI as normal - all calls are tracked automatically!
client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello!"}]
)

That's it! 🎉 View your costs and metrics at app.llmobserve.com

Advanced Usage

Track Agents & Workflows

Automatic Tracking (Zero Config):

from llmobserve import observe
from langchain.agents import AgentExecutor

observe(
    collector_url="https://api.llmobserve.com",
    api_key="llmo_sk_your_key_here"
)

# Agent tracking happens automatically - no manual labeling needed!
agent = AgentExecutor(...)
result = agent.run("query")  # Automatically tracked as agent

Manual Labeling (Optional, for Better UX):

from llmobserve import observe, section

observe(
    collector_url="https://api.llmobserve.com",
    api_key="llmo_sk_your_key_here"
)

# Hierarchical tracking (optional - improves dashboard UX)
with section("agent:research_assistant"):
    with section("tool:web_search"):
        # Your search logic
        results = search_web(query)
    
    with section("tool:summarize"):
        # Your summarization logic
        summary = summarize(results)

Key Points:

• ✅ Costs are always tracked via HTTP interception (works everywhere)
• ✅ Framework agents auto-tracked (LangChain, CrewAI, AutoGen, LlamaIndex)
• ✅ Manual labeling is optional (improves dashboard UX but not required)
• ✅ Untracked costs still shown (labeled as "untracked" in dashboard)

Dashboard shows:

agent:research_assistant    $0.50  (10 calls)
├─ tool:web_search          $0.002  [ok]
└─ tool:summarize           $0.001  [ok]
untracked                   $0.10   (5 calls)  ← Costs without agent labels

Track Costs Per Customer

from llmobserve import observe, set_customer_id

observe(
    collector_url="https://api.llmobserve.com",
    api_key="llmo_sk_your_key_here"
)

# Set customer ID for cost attribution
set_customer_id("customer_xyz")

# All subsequent API calls are attributed to this customer
client = OpenAI()
response = client.chat.completions.create(...)

Decorator Style

from llmobserve import trace

@trace(agent="customer_support")
async def handle_support_request(message: str):
    # All OpenAI calls inside are automatically tracked
    response = await client.chat.completions.create(...)
    return response

Framework Integration

FastAPI

from fastapi import FastAPI
from llmobserve import ObservabilityMiddleware, observe

app = FastAPI()
app.add_middleware(ObservabilityMiddleware)

observe(
    collector_url="https://api.llmobserve.com",
    api_key="llmo_sk_your_key_here"
)

@app.post("/chat")
async def chat(request: Request):
    # Context is auto-reset per request
    # All OpenAI calls are tracked
    response = await client.chat.completions.create(...)
    return response

Flask

from flask import Flask
from llmobserve import flask_before_request, observe

app = Flask(__name__)
app.before_request(flask_before_request)

observe(
    collector_url="https://api.llmobserve.com",
    api_key="llmo_sk_your_key_here"
)

@app.route("/chat")
def chat():
    # Context is auto-reset per request
    response = client.chat.completions.create(...)
    return response

What Gets Tracked

For every LLM API call:

• ✅ Costs - Calculated using latest pricing
• ✅ Tokens - Input, output, and cached tokens
• ✅ Latency - Request duration in milliseconds
• ✅ Status - Success, error, timeout, etc.
• ✅ Model - Which model was used
• ✅ Provider - OpenAI, Anthropic, etc.
• ✅ Section/Agent - Hierarchical context
• ✅ Customer ID - For cost attribution
• ✅ Timestamps - When the call was made
• ⚠️ Prompts/Responses - Optional (disabled by default)

Configuration

Environment Variables

# Disable observability
LLMOBSERVE_DISABLED=1

# Enable content capture (prompts/responses)
ALLOW_CONTENT_CAPTURE=true

Custom Flush Interval

observe(
    collector_url="https://api.llmobserve.com",
    api_key="llmo_sk_your_key_here",
    flush_interval_ms=1000  # Flush every 1 second (default: 500ms)
)

Dashboard Features

Visit app.llmobserve.com to view:

• 💰 Cost Overview - Total spend, trends, by provider/model
• 📊 Run Details - Individual request breakdowns
• 🎯 Agent Analytics - Which agents/workflows cost the most
• 👥 Customer Breakdown - Cost per end-user
• 🚨 Alerts - Cost spikes, errors, anomalies
• 📈 Insights - Auto-detected inefficiencies

Self-Hosting

Want to run your own instance?

git clone https://github.com/yourusername/llmobserve
cd llmobserve

# Start collector API
cd collector
pip install -r requirements.txt
uvicorn main:app --port 8000

# Start dashboard
cd ../web
npm install
npm run dev

See DEPLOYMENT.md for production deployment guide.

Architecture

┌─────────────────┐
│   Your App      │
│  + llmobserve   │  ← Auto-instruments OpenAI & Pinecone
│     SDK         │
└────────┬────────┘
         │ POST /events
         ↓
┌─────────────────┐
│  Collector API  │  ← FastAPI + PostgreSQL
│   (Backend)     │
└────────┬────────┘
         │ GET /runs, /stats
         ↓
┌─────────────────┐
│   Dashboard     │  ← Next.js + Clerk
│   (Frontend)    │
└─────────────────┘

FAQ

Q: Does this slow down my app? A: No. Events are buffered asynchronously with sub-millisecond overhead.

Q: Are my prompts/responses sent to your servers? A: No, by default only metadata is sent (tokens, cost, latency). Set ALLOW_CONTENT_CAPTURE=true to enable content logging.

Q: Can I use this in production? A: Yes! It's production-ready and tested at scale.

Q: What about other LLM providers? A: Anthropic, Cohere, and others coming soon! Request support via GitHub issues.

Q: How much does it cost? A: Free tier includes 10K tracked API calls/month. See pricing.

Contributing

We welcome contributions! See CONTRIBUTING.md.

License

MIT License - see LICENSE

Support

• 📖 Documentation
• 🐛 GitHub Issues
• 💬 Discord Community
• 📧 Email Support

---

Made with ❤️ by the LLM Observe team