modaltrace 0.2.2

OpenTelemetry observability library for real-time AI applications

ModalTrace

OpenTelemetry observability for real-time AI video applications

Docs • API Reference • Examples • Issues • Contributing

---

🚀 What is ModalTrace?

ModalTrace is an open-source OpenTelemetry library that provides production-grade observability for real-time AI video applications. It captures traces, metrics, and logs across your entire ML pipeline—from GPU operations and neural network inference to video rendering and transport layer performance.

Built for production AI systems, ModalTrace helps you understand latency bottlenecks, monitor resource utilization, and debug performance issues in complex distributed video AI pipelines.

---

✨ Features

📊 Comprehensive Observability Trace execution paths across your entire pipeline with automatic span generation for each processing stage. Correlate traces with structured logging and metrics for complete visibility.

⚡ Real-time Metrics Monitor frame rates, GPU memory allocation, inference latency, and synchronization drift with sub-millisecond precision. Built-in ring buffer aggregation for efficient metric collection.

🔍 Semantic Conventions All telemetry uses standardized attribute keys (modaltrace.*) eliminating magic strings and enabling consistent instrumentation across teams.

🧠 PyTorch & GPU Instrumentation Automatic instrumentation of PyTorch operations, GPU memory tracking, and device utilization metrics. Understand exactly where your model is spending time.

🎬 Audio/Video Synchronization Tracking Detect and monitor A/V sync drift with configurable thresholds. Automatically log chunk mismatches and jitter metrics.

🎯 Adaptive Sampling Intelligent span sampling based on anomaly detection. Automatically capture high-latency frames and unusual execution patterns without overwhelming your backend.

🛡️ PII Scrubbing Built-in scrubbing pipeline removes sensitive data from logs and attributes. Customizable patterns for your specific compliance needs.

📡 OpenTelemetry Export Native support for OTLP over HTTP and gRPC. Export to any OpenTelemetry-compatible backend: Jaeger, Datadog, New Relic, Honeycomb, or on-prem observability stacks.

---

🚀 Quick Start

1. Install

pip install modaltrace

For optional features:

pip install modaltrace[pytorch,gpu,webrtc]

2. Configure & Initialize

Create a .env file or set environment variables:

MODALTRACE_SERVICE_NAME=my-video-pipeline
MODALTRACE_OTLP_ENDPOINT=http://localhost:4318
MODALTRACE_PYTORCH_INSTRUMENTATION=true
MODALTRACE_GPU_MONITORING=true

Initialize the SDK in your application:

from modaltrace import ModalTraceSDK

sdk = ModalTraceSDK()
sdk.start()

# Your ML pipeline code here
# Spans are automatically created for instrumented operations

3. View Telemetry

Access your traces at your observability backend (e.g., http://localhost:16686 for Jaeger):

# Start a pipeline span
from modaltrace import get_tracer
tracer = get_tracer(__name__)

with tracer.start_as_current_span("process_frame") as span:
    span.set_attribute("modaltrace.pipeline.frame.sequence_number", frame_id)
    # Your frame processing logic here

---

⚙️ Configuration

All configuration is available via environment variables (prefix: MODALTRACE_) or Python kwargs to ModalTraceConfig:

Setting	Env Variable	Type	Default	Description
Service Identity
service_name	MODALTRACE_SERVICE_NAME	str	modaltrace-pipeline	Service identifier in telemetry
service_version	MODALTRACE_SERVICE_VERSION	str	0.0.0	Semantic version of your service
deployment_environment	MODALTRACE_DEPLOYMENT_ENVIRONMENT	str	development	Environment (dev/staging/prod)
OTLP Export
otlp_endpoint	MODALTRACE_OTLP_ENDPOINT	URL	http://localhost:4318	OpenTelemetry collector endpoint
otlp_protocol	MODALTRACE_OTLP_PROTOCOL	str	http	Protocol: http or grpc
otlp_timeout_ms	MODALTRACE_OTLP_TIMEOUT_MS	int	10000	Export timeout in milliseconds
Feature Flags
pytorch_instrumentation	MODALTRACE_PYTORCH_INSTRUMENTATION	bool	true	Auto-instrument PyTorch ops
gpu_monitoring	MODALTRACE_GPU_MONITORING	bool	true	Track GPU metrics
webrtc_monitoring	MODALTRACE_WEBRTC_MONITORING	bool	false	Monitor WebRTC transports
eventloop_monitoring	MODALTRACE_EVENTLOOP_MONITORING	bool	true	Track event loop blocks
Sampler
pytorch_sample_rate	MODALTRACE_PYTORCH_SAMPLE_RATE	float (0-1)	0.01	Fraction of PyTorch ops to sample
anomaly_threshold_ms	MODALTRACE_ANOMALY_THRESHOLD_MS	float	50.0	Latency threshold for anomaly capture
Metrics
metrics_flush_interval_ms	MODALTRACE_METRICS_FLUSH_INTERVAL_MS	int	1000	Metric aggregation window
ring_buffer_size	MODALTRACE_RING_BUFFER_SIZE	int	512	Must be power of 2
A/V Sync
av_drift_warning_ms	MODALTRACE_AV_DRIFT_WARNING_MS	float	40.0	Warn if drift exceeds this
av_chunk_ttl_s	MODALTRACE_AV_CHUNK_TTL_S	float	5.0	Chunk retention period
PII Scrubbing
scrubbing_enabled	MODALTRACE_SCRUBBING_ENABLED	bool	true	Enable PII removal
scrubbing_patterns	MODALTRACE_SCRUBBING_PATTERNS	list[str]	[]	Regex patterns to scrub

---

📁 Architecture

modaltrace/
├── config.py                 # Pydantic configuration model
├── _registry.py             # Global SDK registry
├── conventions/
│   └── attributes.py        # Semantic convention constants
├── tracing/
│   ├── pipeline.py          # Main trace pipeline
│   ├── sampler.py           # Adaptive sampling logic
│   ├── pending.py           # Pending spans management
│   └── propagation.py       # Context propagation
├── metrics/
│   ├── instruments.py       # Metric instrument definitions
│   ├── aggregator.py        # Ring buffer aggregation
│   └── av_sync.py           # A/V sync metrics
├── instrumentation/
│   ├── pytorch.py           # PyTorch auto-instrumentation
│   ├── gpu.py               # GPU monitoring
│   ├── eventloop.py         # Event loop tracking
│   └── transport.py         # Network transport metrics
├── logging/
│   ├── api.py               # Structured logging API
│   └── scrubber.py          # PII scrubbing pipeline
└── exporters/
    └── setup.py             # OTLP exporter initialization

---

🔧 Development

Setup

git clone https://github.com/arnabdeypolimi/video_ai_telemetry.git
cd video_ai_telemetry
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e ".[dev,all]"

Run Tests

pytest tests/ -v

Lint & Format

ruff check src/ tests/
ruff format src/ tests/

Type Check

mypy src/

---

🤝 Contributing

Contributions are welcome! Please:

1. Open an issue to discuss changes
2. Fork the repository and create a feature branch
3. Submit a pull request with tests for new functionality
4. Ensure all tests pass and code is linted

For detailed guidelines, see CONTRIBUTING.md.

---

📄 License

ModalTrace is licensed under the Apache License 2.0. See LICENSE for details.

---

Questions? Check the documentation, review API reference, or open an issue.