langchain-oso

Oso observability integration for LangChain agents.

Callback handler that automatically captures and sends all LangChain agent events to Oso's observability platform for monitoring, debugging, and security analysis.

Installation

pip install langchain-oso

Or with poetry:

poetry add langchain-oso

Quick Start

from langchain_oso import OsoObservabilityCallback
from langchain.agents import create_openai_tools_agent, AgentExecutor
from langchain_openai import ChatOpenAI

# Create the callback (reads OSO_AUTH_TOKEN from environment)
callback = OsoObservabilityCallback(
    agent_id="my-support-agent"
)

# Add to your agent
llm = ChatOpenAI(model="gpt-4")
agent = create_openai_tools_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, callbacks=[callback])

# Use your agent - all events are automatically captured
result = await agent_executor.ainvoke({"input": "Hello, how can I help?"})

# Clean up
await callback.aclose()

Using Context Manager (Recommended)

async with OsoObservabilityCallback(agent_id="my-agent") as callback:
    agent_executor = AgentExecutor(agent=agent, tools=tools, callbacks=[callback])
    result = await agent_executor.ainvoke({"input": "Hello"})
    # Automatic cleanup when context exits

Configuration

Environment Variables

Set these in your environment or .env file:

# Required: Your Oso authentication token
OSO_AUTH_TOKEN=your-token-here

# Optional: Custom Oso endpoint (defaults to https://cloud.osohq.com/api/events)
OSO_ENDPOINT=https://cloud.osohq.com/api/events

# Optional: Enable/disable observability (defaults to true)
OSO_OBSERVABILITY_ENABLED=true

Constructor Parameters

OsoObservabilityCallback(
    endpoint="https://cloud.osohq.com/api/events",  # Oso endpoint URL
    auth_token="your-token",                         # Oso auth token
    enabled=True,                                    # Enable/disable sending events
    session_id="unique-session-id",                  # Group related conversations
    metadata={"user_id": "123", "env": "prod"},     # Custom metadata for all events
    agent_id="my-agent"                              # Agent identifier
)

All parameters are optional and fall back to environment variables or defaults.

What Gets Captured

The callback automatically captures all LangChain events:

LLM Events

• Model name and configuration
• Prompts sent to the LLM
• Generated responses
• Token usage (prompt, completion, total)
• Errors and failures

Tool Events

• Tool name and description
• Input parameters
• Output/results
• Execution duration (milliseconds)
• Errors and stack traces

Agent Events

• Agent reasoning and thought process
• Tool selection decisions
• Tool input parameters
• Final outputs
• Complete execution flow

Chain Events

• Chain type and name
• Input parameters
• Output values
• Nested chain execution

Execution Summary

At the end of each agent execution, a summary event is sent with:

• Total execution duration
• Number of LLM calls, tool calls, and agent steps
• Total token usage
• Error count
• Complete execution trace

Event Structure

Every event sent to Oso has this structure:

{
  "event_type": "tool.completed",
  "execution_id": "unique-execution-id",
  "session_id": "conversation-session-id",
  "timestamp": "2024-02-15T10:30:45.123Z",
  "agent_id": "my-agent",
  "data": { /* event-specific data */ },
  "metadata": { /* your custom metadata */ }
}

Event Types

• llm.started / llm.completed / llm.error
• tool.started / tool.completed / tool.error
• agent.action / agent.finished
• chain.started / chain.completed / chain.error
• execution.summary - Final summary with all accumulated data

Error Handling

The callback is designed to fail gracefully:

• Network errors or timeouts won't crash your agent
• Failed event sends are logged but don't interrupt execution
• 5-second timeout for all HTTP requests
• Comprehensive error logging for debugging

Logging

The callback uses Python's standard logging. Configure it in your application:

import logging

# See debug info about events being sent
logging.basicConfig(level=logging.DEBUG)

# Or just warnings and errors
logging.basicConfig(level=logging.WARNING)

# Or configure just this library
logging.getLogger("langchain_oso").setLevel(logging.DEBUG)

Examples

Basic Agent with Tools

from langchain_oso import OsoObservabilityCallback
from langchain.agents import create_openai_tools_agent, AgentExecutor
from langchain_openai import ChatOpenAI
from langchain.tools import tool

@tool
def search_orders(customer_id: str) -> str:
    """Search for customer orders."""
    return f"Orders for {customer_id}: ORD001, ORD002"

async def main():
    async with OsoObservabilityCallback(agent_id="support-agent") as callback:
        llm = ChatOpenAI(model="gpt-4o-mini")
        tools = [search_orders]

        agent = create_openai_tools_agent(llm, tools, prompt)
        agent_executor = AgentExecutor(
            agent=agent,
            tools=tools,
            callbacks=[callback]
        )

        result = await agent_executor.ainvoke({
            "input": "Find orders for customer CUST001"
        })

        print(result["output"])

With Custom Metadata

callback = OsoObservabilityCallback(
    agent_id="support-agent",
    session_id="user-session-123",
    metadata={
        "user_id": "user-456",
        "environment": "production",
        "version": "1.2.3"
    }
)

Multiple Agents in Same Session

session_id = str(uuid.uuid4())

# Agent 1
async with OsoObservabilityCallback(
    agent_id="agent-1",
    session_id=session_id
) as callback1:
    result1 = await agent1_executor.ainvoke(...)

# Agent 2 - same session
async with OsoObservabilityCallback(
    agent_id="agent-2",
    session_id=session_id
) as callback2:
    result2 = await agent2_executor.ainvoke(...)

Requirements

• Python 3.10+
• langchain-core >= 0.1.0
• httpx >= 0.24.0

License

Apache License 2.0

Support

• Documentation: https://www.osohq.com/docs
• Issues: https://github.com/osohq/langchain-oso/issues
• Website: https://www.osohq.com