Lode Observe AI: Agent Observability & Tracing

Deep execution tracer for AI agents. Export traces to Jaeger, DataDog, or any OTEL-compatible backend. Includes cost analytics, anomaly detection, and production observability.

The Problem

Multi-agent systems are black boxes:

• ❌ No visibility into intermediate steps when agents fail or loop
• ❌ Token costs explode without warning
• ❌ Traditional APM tools fail—they're built for deterministic RPCs, not probabilistic token generation
• ❌ No explanation for why the model entered a runaway loop

The Solution

Lode Observe AI maps every step of agent execution with zero overhead (<0.1ms per span). Track:

• Every prompt, token, and decision with hierarchical DAGs
• Cost attribution by model, provider, and agent step
• Anomaly detection for infinite loops, context exhaustion, and runaway spend
• Real-time debugging via Jaeger, DataDog, or local CLI

Installation

# Via pip
pip install lode-observe-ai

# Via Docker
docker run -p 3000:3000 ghcr.io/craftedwithintent/lode-observe-ai:0.1.3.dev0

Quick Start

1. Analyze Traces Locally

from lode_observe_ai import build_trace_tree, aggregate_costs, detect_anomalies

# Build trace tree from spans
tree = build_trace_tree(spans, root_span_id="root_id")

# Analyze costs
costs = aggregate_costs(tree)
print(f"Total tokens: {costs.total_tokens}")
print(f"Total cost: ${costs.total_cost_usd:.4f}")

# Detect anomalies
anomalies = detect_anomalies(tree)
if anomalies.estimated_infinite_loop:
    print(f"⚠️  Infinite loop detected! {anomalies.max_repeated_tool_calls} repeated calls")

2. Export to Jaeger

# Start Jaeger (Docker)
docker run -d -p 6831:6831/udp -p 16686:16686 jaegertracing/all-in-one

# Export trace
lode-observe-ai export --trace-id <trace-id> --format otel --otel-endpoint http://localhost:4317

# Open browser: http://localhost:16686

3. Export to Test Suite

# Convert failed trace into test case
lode-observe-ai export --trace-id abc123 --format assay --output suite.yaml

Key Features

What's Included (Phase 1 MVP)

✅ Core trace analysis (DAG, costs, anomalies)
✅ OpenTelemetry export (Jaeger, DataDog compatible)
✅ CLI tools (export, inspect, analyze)
✅ Docker & Kubernetes deployment
✅ SQLite storage backend
✅ <0.1ms instrumentation overhead

🟡 Phase 2 (coming): Cloud sync, sampling, cycle interceptor
🟡 Phase 3 (coming): Enterprise auth, multi-tenancy, analytics

Configuration

Python Context Manager API

from lode_observe_ai import start_trace, trace_agent

@trace_agent(name="ResearchAgent", sample_rate=1.0)
def my_agent(topic: str) -> str:
    with start_trace("step") as tracer:
        result = do_work(topic)
        tracer.record_output(result)
    return result

Environment Variables

# Storage
LODE_STORAGE=sqlite:///./traces.db

# OTEL export
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_SERVICE_NAME=my-agent

# Instrumentation
LODE_SAMPLE_RATE=1.0  # Trace every request
LODE_OVERHEAD_BUDGET_MS=0.5  # Max overhead allowed

Performance

See the GitHub repo for full benchmarks.

Documentation

• Full README — Comprehensive guide
• Deployment Guide — Docker, Kubernetes, security
• Troubleshooting — Common issues, debug mode
• Performance Guide — Benchmarks, tuning, scaling
• Contributing — Development setup, code standards
• Examples — Research agent with tracing

Support

• 🐛 GitHub Issues — Bug reports, feature requests
• 💬 GitHub Discussions — Questions, ideas
• 📖 Full Repository — Source, examples, docs

License

MIT License — See LICENSE for details.

Completely decoupled: Works standalone. No shared dependencies with other packages. Part of the CraftedWithIntent ecosystem for production AI systems.