AI Monitor - Plug & Play AI Agent Monitoring

A comprehensive, zero-configuration monitoring solution for AI agents that requires no source code changes. Simply import and use decorators or context managers.

🚀 Quick Start

Installation (No Cloning Required!)

One command installs everything:

pip install ai-monitor

That's it! No cloning, no source code changes, no complex setup. Your AI monitoring is ready to use immediately.

Basic Usage (Plug & Play)

1. Ultra Simple - One Line Setup

from ai_monitor import ultra_simple_setup

# Add this ONE line to enable comprehensive monitoring
ultra_simple_setup()

# Your existing AI code works unchanged!
import openai
response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)
# 🎉 Automatically monitored: latency, tokens, quality, traces

2. Enterprise Setup with Traceloop

from ai_monitor import setup_with_traceloop

# Enable enterprise-grade tracing
setup_with_traceloop("my_app_name")

# Your AI code - now with distributed tracing!

3. Flask App Integration

from flask import Flask
from ai_monitor import flask_plug_and_play

app = Flask(__name__)

# One line to monitor entire Flask app
flask_plug_and_play(app)

# All your routes are now monitored automatically!

What You Get Immediately

After pip install ai-monitor, you get:

• ✅ HTTP Interception - Automatic OpenAI API monitoring
• ✅ Prometheus Metrics - Real-time metrics at http://localhost:8000/metrics
• ✅ Quality Analysis - Hallucination detection & drift analysis
• ✅ OpenTelemetry Traces - Ready for Jaeger/Traceloop integration
• ✅ Zero Code Changes - Works with your existing AI applications

1. Decorator Style (Easiest)

from ai_monitor import monitor_llm_call, monitor_agent, monitor_tool_use

# Monitor LLM calls automatically
@monitor_llm_call(model="gpt-4")
def call_openai(prompt):
    # Your existing LLM code - no changes needed!
    response = openai.ChatCompletion.create(...)
    return response

# Monitor agent sessions  
@monitor_agent(name="my_assistant")
def run_agent():
    # Your agent code
    return agent_result

# Monitor tool usage
@monitor_tool_use(tool_name="web_search")
def search_web(query):
    # Your tool code
    return search_results

2. Context Manager Style

from ai_monitor import monitor_llm_call, monitor_agent_session

# Monitor individual LLM calls
with monitor_llm_call("gpt-4", "Hello world") as monitor:
    response = call_llm("Hello world")
    monitor.record_response(
        response="Hello! How can I help?",
        input_tokens=2,
        output_tokens=5,
        cost=0.001
    )

# Monitor complete agent sessions
with monitor_agent_session("customer_service") as session:
    # Record LLM calls within session
    session.record_llm_call("gpt-4", "Help request", "I can help with that", 10, 15, 0.002)
    
    # Record tool usage
    session.record_tool_use("knowledge_base", "search query", "results", success=True, duration=0.5)

3. Global Monitor Instance

from ai_monitor import get_monitor

# Get the default monitor instance
monitor = get_monitor()

# Record data manually
call_id = monitor.record_llm_call(
    model="gpt-3.5-turbo",
    prompt="What is AI?",
    response="AI is artificial intelligence...",
    input_tokens=4,
    output_tokens=20,
    latency=1.2,
    cost=0.001
)

# Get monitoring data
stats = monitor.get_summary_stats()
metrics = monitor.get_metrics()
recent_calls = monitor.get_llm_calls(limit=10)

📊 What Gets Monitored

All OpenAI API Endpoints (Enhanced Coverage)

• Chat Completions: client.chat.completions.create() ✅
• Text Completions: client.completions.create() ✅
• Embeddings: client.embeddings.create() ✅
• Image Generation: client.images.generate() (DALL-E) ✅
• Audio Transcription: client.audio.transcriptions.create() (Whisper) ✅
• Audio Translation: client.audio.translations.create() (Whisper) ✅
• Content Moderation: client.moderations.create() ✅
• Model Operations: client.models.list(), file operations, fine-tuning ✅
• Azure OpenAI: All deployment endpoints with enhanced model detection ✅

Automatic Metrics (All Endpoints)

• Latency & Performance: Response time, tokens per second, throughput
• Token Usage: Input/output tokens, token costs, efficiency ratios
• Quality Indicators: Response completeness, structure, consistency
• Error Tracking: Failure rates, error types, recovery times
• Resource Usage: Memory, CPU (when psutil installed)

Advanced Detection

• Hallucination Detection: Pattern-based detection of uncertain or fabricated content
• Model Drift: Automatic detection of performance degradation over time
• Cost Monitoring: Real-time cost tracking and budgeting

Trace & Context

• Request Tracing: End-to-end request flows with Jaeger integration
• Agent Sessions: Multi-step agent conversations and decision paths
• Tool Usage: External API calls and tool performance

📈 Data Export Options

Built-in Exporters

• Prometheus: Metrics collection with built-in HTTP server
• Jaeger: Distributed tracing for request flows
• Structured Logs: JSON logs for analysis
• Console: Real-time monitoring output
• JSON Files: Local data storage

Configuration Example

from ai_monitor import init_monitoring, MonitoringConfig

config = MonitoringConfig(
    # Exporters
    enable_prometheus=True,
    enable_jaeger=True, 
    enable_logging=True,
    
    # Prometheus
    prometheus_port=8000,
    
    # Features
    detect_hallucination=True,
    detect_drift=True,
    track_costs=True,
    
    # Sampling
    trace_sampling_rate=1.0
)

monitor = init_monitoring(config)

🔍 Monitoring Features

Hallucination Detection

from ai_monitor.detectors import HallucinationDetector

detector = HallucinationDetector()
result = detector.detect(
    prompt="What is the capital of France?",
    response="I think it might be Paris, but I'm not entirely sure...",
    context="France is a country in Europe."
)

print(f"Is hallucination: {result.is_hallucination}")
print(f"Confidence: {result.confidence_score}")
print(f"Reasons: {result.reasons}")

Drift Detection

from ai_monitor.detectors import DriftDetector

drift_detector = DriftDetector()

# Update with each LLM call
drift_detector.update(
    latency=1.2,
    input_tokens=10,
    output_tokens=50,
    cost=0.002,
    response="AI response text"
)

# Check for drift
drift_results = drift_detector.detect_drift()
for drift in drift_results:
    if drift.has_drift:
        print(f"Drift detected in {drift.drift_type}")
        print(f"Baseline: {drift.baseline_value}, Current: {drift.current_value}")

📋 Dashboard & Visualization

Prometheus Metrics (localhost:8000)

ai_monitor_llm_calls_total{model="gpt-4"} 150
ai_monitor_llm_latency_seconds_bucket{model="gpt-4",le="1.0"} 120
ai_monitor_tokens_total{model="gpt-4",type="input"} 5000
ai_monitor_tokens_total{model="gpt-4",type="output"} 15000
ai_monitor_cost_total{model="gpt-4"} 2.50

Getting Summary Statistics

from ai_monitor import get_monitor

monitor = get_monitor() 
stats = monitor.get_summary_stats()

print(f"Total LLM calls: {stats['total_calls']}")
print(f"Total tokens: {stats['total_tokens']}")
print(f"Total cost: ${stats['total_cost']:.4f}")
print(f"Average latency: {stats['average_latency']:.2f}s")
print(f"Tokens per second: {stats['tokens_per_second']:.1f}")

🛠️ Integration Examples

OpenAI Integration

from ai_monitor import monitor_llm_call
import openai

@monitor_llm_call()  # Automatically detects OpenAI format
def chat_with_gpt(messages):
    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=messages
    )
    return response

# Usage - monitoring happens automatically!
result = chat_with_gpt([{"role": "user", "content": "Hello"}])

Anthropic Claude Integration

@monitor_llm_call(model="claude-3")
def chat_with_claude(prompt):
    response = anthropic.completions.create(
        model="claude-3-sonnet-20240229",
        prompt=prompt,
        max_tokens_to_sample=100
    )
    return response

Custom Agent Integration

@monitor_agent(name="research_assistant")
def research_agent(query):
    with monitor_agent_session("research_session") as session:
        
        # Search step
        search_results = search_web(query)
        session.record_tool_use("web_search", query, search_results, True, 2.1)
        
        # Analysis step
        analysis = analyze_with_llm(search_results)
        session.record_llm_call("gpt-4", f"Analyze: {search_results}", analysis, 100, 200, 0.01)
        
        # Summary step
        summary = summarize_findings(analysis)
        session.record_llm_call("gpt-3.5-turbo", f"Summarize: {analysis}", summary, 200, 50, 0.002)
        
        return summary

📊 Grafana Dashboard Setup

1. Install Grafana and Prometheus
2. Configure Prometheus to scrape localhost:8000/metrics
3. Import dashboard with these key metrics:
   • LLM call rate and latency
   • Token usage and costs
   • Error rates and drift detection
   • Agent session success rates

🚨 Alerting

Prometheus Alerting Rules

groups:
- name: ai_monitor_alerts
  rules:
  - alert: HighLatency
    expr: ai_monitor_llm_latency_seconds > 5
    labels:
      severity: warning
    annotations:
      summary: "High LLM latency detected"
      
  - alert: CostThreshold
    expr: increase(ai_monitor_cost_total[1h]) > 10
    labels:
      severity: critical
    annotations:
      summary: "Hourly cost exceeds $10"

🔧 Advanced Configuration

Custom Exporters

from ai_monitor import configure_exporters, get_monitor

monitor = get_monitor()
configure_exporters(monitor, [
    {'type': 'console'},
    {'type': 'json_file', 'file_prefix': 'my_app_monitor'}
])

Performance Profiling

from ai_monitor.utils import PerformanceProfiler

profiler = PerformanceProfiler()

with profiler.profile_operation("llm_inference"):
    result = call_llm("prompt")

summary = profiler.get_performance_summary()

🎯 Key Benefits

• Zero Code Changes: Drop-in decorators and context managers
• Comprehensive Monitoring: Tracks everything from tokens to traces
• Production Ready: Prometheus, Jaeger, and structured logging
• Intelligent Detection: Automated hallucination and drift detection
• Cost Awareness: Real-time cost tracking and optimization
• Performance Insights: Detailed latency and throughput analysis

📝 License

MIT License - feel free to use in your projects!

Ready to monitor your AI agents? Just pip install the dependencies and add a single decorator! 🚀