🚀 OTEL-Lean-Tracer
🎯 Zero-config OpenTelemetry tracing with Lean Production KPIs for LLM/Agent applications
OTEL-Lean-Tracer seamlessly adds "Lean Production 7 Wastes" metrics to your LLM and Agent execution traces, giving you real-time visibility into bottlenecks and waste in your AI applications through beautiful Grafana dashboards.
✨ Why OTEL-Lean-Tracer?
🔍 Instant Visibility
- Zero configuration - Just add one line and start seeing your AI application's performance
- Real-time bottleneck detection - Spot queue delays, context waste, and retry loops instantly
- Beautiful Grafana dashboards - Pre-built visualizations for immediate insights
📊 Lean Production Metrics
Track the 7 wastes of Lean Production applied to LLM applications:
- 🕒 Queue Time - Time spent waiting for API responses
- 🔄 Retry Waste - Failed generation attempts and retries
- 📈 Context Waste - Unused tokens in prompts
- 💰 Cost Tracking - Real-time API cost monitoring (generation + evaluation)
- ⚡ Processing Delays - Identify slow components
- 🎯 Quality Issues - Track acceptance rates and evaluation scores
- 🚚 Transport Overhead - Network and serialization costs
🎯 GenAgent Evaluation Tracking
Advanced quality monitoring for agents-sdk-models:
- 📊 Evaluation Scores - Track quality scores (0-100) vs thresholds
- 🔄 Retry Analysis - Monitor threshold-based retry patterns
- 💡 Improvement Insights - Capture AI-generated improvement suggestions
- 💰 Separate Cost Tracking - Split generation and evaluation costs
- 📈 Quality Trends - Visualize quality progression over time
- 🎛 Multi-Step Monitoring - Track quality across Flow pipelines
🛠 Developer Friendly
- OpenTelemetry compatible - Works with your existing observability stack
- Agent SDK integration - Built for modern LLM frameworks
- Grafana + Tempo ready - Enterprise-grade trace storage and visualization
🚀 Quick Start
Installation
# Install via uv (recommended)
uv add otel-lean-tracer
# Or via pip
pip install otel-lean-tracer
⚡ Super Simple Setup
Add just one line to your LLM application:
from otel_lean_tracer import instrument_lean
# Enable lean tracing (that's it!)
instrument_lean()
# Your existing code works unchanged
from agents_sdk_models import create_simple_gen_agent
agent = create_simple_gen_agent(
name="story_generator",
instructions="Generate creative stories",
model="gpt-4o-mini",
threshold=75 # Quality threshold for evaluation
)
result = await agent.run("Tell me a story about a robot")
📊 Instant Dashboard
Your traces automatically include:
Core Attributes (OTEL Semantic Conventions):
otel.lean.gen.attempt- Generation attempt number (1, 2, 3...)otel.lean.gen.accepted- Whether output was acceptedllm.evaluation.score- Quality evaluation score (0-100)llm.evaluation.threshold- Quality threshold settingllm.evaluation.passed- Whether evaluation passedotel.lean.retry.reason- Why retry was neededllm.context.bytes_total- Total prompt sizellm.context.bytes_useful- Actually referenced contentotel.lean.queue_age_ms- Time waiting in queuesllm.usage.cost_usd- Generation API costsllm.evaluation.cost_usd- Evaluation API costs
Optional Attributes (when available):
model_scale_b- Model scale in billions of parametersretry_energy_kwh- Estimated energy consumption for retries
🎛 Supported Environments
| Environment | Status | Version |
|---|---|---|
| Python | ✅ | 3.11+ |
| OpenTelemetry | ✅ | 1.25+ |
| Agents SDK | ✅ | 0.22+ |
| Grafana Tempo | ✅ | Latest |
| OpenAI | ✅ | All models |
| Anthropic | ✅ | Claude series |
| Ollama | ✅ | Local models |
📈 Advanced Configuration
Custom Exporters
from otel_lean_tracer import instrument_lean
from otel_lean_tracer.exporters import create_tempo_exporter
# Configure Tempo backend
exporter = create_tempo_exporter(
endpoint="http://tempo:3200",
headers={"Authorization": "Bearer your-token"}
)
instrument_lean(exporter=exporter)
Evaluation Data Tracking
from otel_lean_tracer.attributes import LeanAttributes
# Evaluation data is automatically captured from GenAgent:
# - Quality scores and thresholds
# - Improvement suggestions
# - Retry reasons and patterns
# - Separate cost tracking for generation vs evaluation
Flow Metrics
from otel_lean_tracer.mixins import FlowInstrumentationMixin
class MyAgent(FlowInstrumentationMixin):
async def process(self, input_data):
# Automatically measures queue_age_ms and evaluation metrics
return await self.instrumented_call(input_data)
🎯 Real-World Benefits
💡 Before OTEL-Lean-Tracer
- ❌ No visibility into LLM application performance
- ❌ Manual cost tracking and estimation
- ❌ Difficult to spot bottlenecks and waste
- ❌ No quality monitoring or evaluation insights
- ❌ Complex observability setup required
✨ After OTEL-Lean-Tracer
- ✅ Zero-config observability for LLM apps
- ✅ Automatic cost tracking with real-time dashboards
- ✅ Instant bottleneck detection using Lean metrics
- ✅ Quality evaluation monitoring with improvement insights
- ✅ Beautiful Grafana dashboards out of the box
🔧 Integration Examples
With OpenAI Agents SDK + Evaluation
from agents_sdk_models import create_simple_gen_agent, Flow
from otel_lean_tracer import instrument_lean
instrument_lean()
# Create agents with quality thresholds
classifier = create_simple_gen_agent(
name="intent_classifier",
instructions="Classify user intent accurately",
model="gpt-4o-mini",
threshold=80 # Higher threshold for classification
)
responder = create_simple_gen_agent(
name="response_generator",
instructions="Generate helpful responses",
model="gpt-4o",
threshold=75 # Quality threshold for responses
)
# Create flow - evaluation data automatically traced!
flow = Flow(steps=[classifier, responder])
result = await flow.run("Help me plan a vacation")
# Dashboard shows:
# - Quality scores for each step
# - Cost breakdown (generation vs evaluation)
# - Retry patterns and improvement suggestions
# - Multi-step quality progression
With Custom Evaluation Systems
from otel_lean_tracer.processors import LeanTraceProcessor
from opentelemetry import trace
# Custom evaluation data is automatically detected
custom_result = {
"content": "Generated response",
"evaluation": {
"score": 85.5,
"threshold": 75.0,
"passed": True,
"improvements": ["Add more examples", "Improve clarity"]
}
}
# Lean processor extracts evaluation data automatically
processor = LeanTraceProcessor()
trace.get_tracer_provider().add_span_processor(processor)
📊 Grafana Dashboard Setup
Pre-built dashboards included in /examples/grafana/:
- Lean Production Metrics - Overview of waste and efficiency
- Quality Analysis - Evaluation scores, trends, and improvement tracking
- Cost Analysis - Real-time API cost breakdown (generation + evaluation)
- Performance Monitoring - Latency and throughput metrics
- Error Analysis - Retry patterns and failure modes
Import lean-tracer-dashboard.json into your Grafana instance for instant visualizations!
🏗 Architecture
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Your LLM App │───▶│ OTEL-Lean-Tracer │───▶│ Grafana + Tempo │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│
▼
┌──────────────┐
│ Lean Metrics │
│ • Queue Time │
│ • Retry Waste│
│ • Context │
│ • Quality │
│ • Cost │
└──────────────┘
🤝 Contributing
We love contributions! Check out our Contributing Guide to get started.
Development Setup
# Clone the repository
git clone https://github.com/your-org/otel-lean-tracer.git
cd otel-lean-tracer
# Install in development mode
uv pip install -e .
# Install dev dependencies
uv add --group dev pytest pytest-cov pytest-asyncio
# Run tests
pytest
📚 Documentation
- 📖 Complete Documentation
- 🏗 Architecture Guide
- 🎯 Use Cases
- 🔧 Function Reference
- 📊 Grafana Setup Guide
- 🎓 Tutorial with Evaluation Examples
📄 License
MIT License - see LICENSE file for details.
🙏 Acknowledgments
- OpenTelemetry community for the amazing observability framework
- Grafana team for beautiful visualization tools
- Lean Production methodology for waste reduction principles
- agents-sdk-models for advanced LLM agent capabilities
Made with ❤️ for the LLM/Agent developer community
Start tracking your AI application's efficiency AND quality in under 60 seconds! 🚀
Metadata
Release files for otel-lean-tracer 0.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| otel_lean_tracer-0.0.1.tar.gz | 41.8 kB | Without provenance | Details for otel_lean_tracer-0.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| otel_lean_tracer-0.0.1-py3-none-any.whl | Python 3 | none | any | Details for otel_lean_tracer-0.0.1-py3-none-any.whl |
Total release size: 73.8 kB
Release files / otel_lean_tracer-0.0.1.tar.gz
| Download URL | otel_lean_tracer-0.0.1.tar.gz |
|---|---|
| Size | 41.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
09543352b6d6f7534bb9b27218610cddc80f0f932b1dfbc109f092c4136e7f8c
|
|
BLAKE2b-256 checksum How to use checksums |
71b721e7a343763afdd7c663c38abb4637d467d70ed099c454d27f8ba7fd6265
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.12
|
Release files / otel_lean_tracer-0.0.1-py3-none-any.whl
| Download URL | otel_lean_tracer-0.0.1-py3-none-any.whl |
|---|---|
| Size | 32.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
90f50c8f42232db199388c53dabe44d1ae0871dfd98a97e55c0d659e1ff2b139
|
|
BLAKE2b-256 checksum How to use checksums |
10cbb2e59b035f414eacb750866fe7ce46ebbc64ea706579e89503a823c005a9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.12
|
Security for otel-lean-tracer 0.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.