TraceVerde

The most comprehensive OpenTelemetry auto-instrumentation library for LLM/GenAI applications

Trace from OpenTelemetry traces. Verde meaning green - for sustainable, transparent AI observability.

Documentation | Examples | Discord | PyPI

Get Started in 30 Seconds

pip install genai-otel-instrument

import genai_otel
genai_otel.instrument()

# Your existing code works unchanged - traces, metrics, and costs are captured automatically
import openai
client = openai.OpenAI()
response = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "Hello!"}])

That's it. No wrappers, no decorators, no config files. Every LLM call, database query, and agent interaction is automatically traced with full cost breakdown.

Why TraceVerde?

Feature	TraceVerde	OpenLIT	Traceloop/OpenLLMetry	Langfuse	Galileo	Arize (Phoenix)	Opik (Comet)
Zero-code setup	Yes	Yes	Yes	SDK required	SDK required	SDK / auto (OpenInference)	SDK required
LLM providers	26	25+	15+	Via integrations	Via integrations	Via integrations	Via integrations
Multi-agent frameworks	8 (CrewAI, LangGraph, ADK, AutoGen, OpenAI Agents, Pydantic AI, etc.)	Limited	Limited	Limited	Limited	Limited	Limited
Cost tracking	Automatic (1,700+ models)	Manual config	Manual config	Manual config	Yes	Yes	Yes
GPU metrics (NVIDIA + AMD)	Yes	No	No	No	No	No	No
MCP tool instrumentation	Yes (databases, caches, vector DBs, queues)	Limited	Limited	No	No	No	No
Evaluation (PII, toxicity, bias, hallucination, prompt injection)	Built-in (6 detectors)	No	No	Separate service	Extensive (core focus)	Built-in (Phoenix evals)	Built-in (core focus)
OpenTelemetry native	Yes	Yes	Yes	Partial	Partial	Yes (OpenInference)	Partial
Self-hosted / on-prem	Yes (fully local)	Yes	Yes	Yes	Enterprise tier	Yes (Phoenix)	Yes
License	Apache-2.0	Apache-2.0	Apache-2.0	MIT	Proprietary	Open-source + Commercial	Apache-2.0

What Gets Instrumented?

LLM Providers (26+)

OpenAI, OpenRouter, CometAPI, Anthropic, Google AI, Google GenAI, AWS Bedrock, Azure OpenAI, Cohere, Mistral AI, Together AI, Groq, Ollama, vLLM, llama.cpp, Vertex AI, Replicate, HuggingFace, SambaNova, Sarvam AI, ElevenLabs, Hyperbolic, LiteLLM, TypeSafe AI / Jev

See all providers with examples >>

Multi-Agent Frameworks (8)

CrewAI, LangGraph, Google ADK, AutoGen, AutoGen AgentChat, OpenAI Agents SDK, Pydantic AI, AWS Bedrock Agents

See all frameworks with examples >>

MCP Tools (20+)

Databases: PostgreSQL, MySQL, MongoDB, SQLAlchemy, TimescaleDB, OpenSearch, Elasticsearch, FalkorDB Caching: Redis | Queues: Kafka, RabbitMQ | Storage: MinIO Vector DBs: Pinecone, Weaviate, Qdrant, ChromaDB, Milvus, FAISS, LanceDB

See all MCP tools >>

Built-in Evaluation (6 Detectors)

PII Detection (GDPR/HIPAA/PCI-DSS), Toxicity Detection, Bias Detection, Prompt Injection Detection, Restricted Topics, Hallucination Detection

See all evaluation features with examples >>

Screenshots

OpenAI traces with token usage, costs, and latency

More screenshots

Ollama (Local LLM)

SmolAgents with Tool Calls

GPU Metrics

OpenSearch Dashboard

Key Features

Automatic Cost Tracking

1,700+ models across 40+ providers with per-request cost breakdown. Supports differential pricing (prompt vs completion), reasoning tokens, cache pricing, and custom model pricing.

# Cost tracking is enabled by default - just instrument and go
genai_otel.instrument()

# Or add custom pricing for proprietary models
export GENAI_CUSTOM_PRICING_JSON='{"chat":{"my-model":{"promptPrice":0.001,"completionPrice":0.002}}}'

Cost tracking guide >>

GPU Metrics (NVIDIA + AMD)

Real-time monitoring of utilization, memory, temperature, power, PCIe throughput, throttling, and ECC errors. Multi-GPU aggregate metrics included.

pip install genai-otel-instrument[gpu]      # NVIDIA
pip install genai-otel-instrument[amd-gpu]  # AMD

GPU metrics guide >>

Multi-Agent Tracing

Complete span hierarchy for agent frameworks with automatic context propagation:

Crew Execution
  +-- Agent: Senior Researcher (gpt-4)
  |     +-- Task: Research OpenTelemetry
  |           +-- openai.chat.completions (tokens: 1250, cost: $0.03)
  +-- Agent: Technical Writer (ollama:llama2)
        +-- Task: Write blog post
              +-- ollama.chat (tokens: 890, cost: $0.00)

Multimodal Observability (v1.1.0)

First-class capture of image, audio, video, and document content parts on OpenAI, Anthropic, Google Gemini, and Groq spans. Bytes are offloaded to your configured object store (MinIO / S3 / filesystem / HTTP) and referenced from spans by URI — they never appear inline in span attributes.

# Opt in (default is off — text-only behaviour is byte-identical to 1.0.x)
export GENAI_OTEL_MEDIA_CAPTURE_MODE=full
export GENAI_OTEL_MEDIA_STORE=minio
export GENAI_OTEL_MEDIA_STORE_ENDPOINT=http://localhost:9000
export GENAI_OTEL_MEDIA_STORE_ACCESS_KEY=...
export GENAI_OTEL_MEDIA_STORE_SECRET_KEY=...
# Optional: plug in a redactor before upload
export GENAI_OTEL_MEDIA_REDACTOR=genai_otel.media.redactors.face_blur

Spans get two co-emitted representations of the same multimodal content:

A flat, queryable attribute namespace — gen_ai.prompt.{n}.content.{m}.{type, media_uri, media_mime_type, media_byte_size, media_source} plus a gen_ai.completion.* mirror — for backends that index on flat attributes.
The upstream-canonical gen_ai.input.messages / gen_ai.output.messages JSON conforming to the gen-ai message schemas in the dedicated semantic-conventions-genai repo, including the document modality standardised by our upstream PR #142, plus the optional byte_size and stripped_reason shapes proposed in #143 / #144 (see Standards Contributions below).

Multimodal guide >>

Safety & Evaluation

genai_otel.instrument(
    enable_pii_detection=True,       # GDPR/HIPAA/PCI-DSS compliance
    enable_toxicity_detection=True,  # Perspective API + Detoxify
    enable_bias_detection=True,      # 8 bias categories
    enable_prompt_injection_detection=True,
    enable_hallucination_detection=True,
    enable_restricted_topics=True,
)

Evaluation guide with 50+ examples >>

Configuration

# Required
export OTEL_SERVICE_NAME=my-llm-app
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# Optional
export GENAI_ENABLE_GPU_METRICS=true
export GENAI_ENABLE_COST_TRACKING=true
export GENAI_SAMPLING_RATE=0.5                    # Reduce volume in production
export GENAI_ENABLED_INSTRUMENTORS=openai,crewai  # Select specific instrumentors

Full configuration reference >>

Backend Integration

Works with any OpenTelemetry-compatible backend:

Jaeger, Zipkin, Prometheus, Grafana, Datadog, New Relic, Honeycomb, AWS X-Ray, Google Cloud Trace, Elastic APM, Splunk, SigNoz, self-hosted OTel Collector

Pre-built Grafana dashboard templates included.

Examples

90+ ready-to-run examples covering every provider, framework, and evaluation feature:

examples/
+-- openai/              # OpenAI chat, embeddings
+-- anthropic/           # Anthropic + PII/toxicity detection
+-- ollama/              # Local models + all evaluation features
+-- crewai_example.py    # Multi-agent crew orchestration
+-- langgraph_example.py # Stateful graph workflows
+-- google_adk_example.py # Google Agent Development Kit
+-- autogen_example.py   # Microsoft AutoGen agents
+-- pii_detection/       # 10 PII examples (GDPR, HIPAA, PCI-DSS)
+-- toxicity_detection/  # 8 toxicity examples
+-- bias_detection/      # 8 bias examples (hiring compliance, etc.)
+-- prompt_injection/    # 6 injection defense examples
+-- hallucination/       # 4 hallucination detection examples
+-- ...                  # And many more

Browse all examples >>

OpenTelemetry Standards Contributions

TraceVerde isn't only a consumer of OpenTelemetry GenAI semantic conventions — production gaps surfaced by the library are being upstreamed back into the spec, and one is now part of the standard. Proposals on open-telemetry/semantic-conventions-genai:

PR	Proposal	Status
#142	Add `document` to the `Modality` enum on `BlobPart` / `FilePart` / `UriPart` — PDFs, DOCX, and other non-image/video/audio payloads currently fall through to the free-form `string` branch. BFSI KYC extraction is a high-volume real example.	Merged 2026-05-16
#143	Add optional `byte_size` on the three media-part types so consumers get a uniform handle on payload size whether the content was carried inline, by URI, or by provider file id — useful for cost-of-capture telemetry and storage planning. Pydantic `ge=0` → JSON schema `"minimum": 0`.	Open — reference scenarios for `FilePart` / `UriPart` requested
#144	Make `content` / `file_id` / `uri` optional and add a free-form `stripped_reason` (`size_exceeded`, `modality_not_allowed`, `redactor_error`, `upload_error`, `no_store_configured`) so an instrumentation can fail-closed — record that it observed a media part but intentionally did not capture its bytes — while preserving the original part `type` and `modality`. Enforced via a top-level `anyOf` so structurally-empty parts cannot validate.	Open — with reviewers

All three were migrated from the closed open-telemetry/semantic-conventions#3673 after the GenAI conventions split into the dedicated repo on 2026-05-05. Each PR ships under the new repo's V2 Weaver schema with the corresponding models.ipynb updates and make check-policies / make generate-all validation.

TraceVerde already emits this shape on the wire, providing the reference implementation for these conventions. Since v1.10.0 the GenAI semconv tier is opted into by default (OTEL_SEMCONV_STABILITY_OPT_IN defaults to gen_ai/dup), so spans carry both the current and the superseded attribute spellings.

Ecosystem & Framework Contributions

Beyond the spec, genai_otel is shipping inside agent frameworks as their OpenTelemetry observability layer — in-tree where the project accepts it, or as a standalone plugin where the project's policy keeps integrations out of the core tree — demonstrating the library powering real third-party agents, not just first-party services:

Integration	Framework	Contribution	Status
hermes-otel-plugin	NousResearch Hermes	A standalone `otel` plugin (`hermes plugins install Mandark-droid/hermes-otel-plugin`) exporting Hermes turns / LLM calls / tool calls as OTel GenAI spans and dashboard log records, with no changes to Hermes core. When `genai-otel-instrument` is installed it additionally unlocks on-prem GPU / energy / CO2 metrics, local-model cost (parameter-size pricing for Ollama / HF / vLLM), and an inline eval / guardrail suite (PII, toxicity, bias, prompt-injection, restricted-topics, hallucination) scored on prompt and response — signals no vanilla OTel SDK or other GenAI instrumentor emits. Originally upstream PR #48184 — approved on code review, then republished standalone per Hermes's policy that observability backends ship as standalone plugin repos.	Shipped (standalone plugin, MIT)

Who Uses TraceVerde?

TraceVerde is used by developers and teams building production GenAI applications. If you're using TraceVerde, we'd love to hear from you!

Add your company | Join Discord

Community

Documentation - Full guides, API reference, and tutorials
Discord - Chat with the community
GitHub Issues - Bug reports and feature requests
Contributing - How to contribute
Human and LLM agent contributions - How agents can report gaps or submit focused fixes

License

openinference typesafe strands http evaluation gpu amd-gpu all-gpu co2 openai openrouter cometapi anthropic google aws azure azure-ai-inference cohere mistral together groq llamacpp ollama vllm replicate sambanova sarvamai elevenlabs liquid-audio langchain langgraph llamaindex huggingface sentence-transformers crewai autogen autogen-agentchat google-adk haystack dspy guardrails instructor pydantic-ai vertexai databases messaging falkordb vector-dbs object-storage multimodal-images multimodal-pdf multimodal-faces multimodal-s3 multimodal all-providers all-mcp all dev

Release files for genai-otel-instrument 1.30.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for genai-otel-instrument 1.30.0
File	Size	Uploaded
genai_otel_instrument-1.30.0.tar.gz	2.8 MB	Sep 24, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for genai-otel-instrument 1.30.0
File	Interpreter	ABI	Platform	Reset
genai_otel_instrument-1.30.0-py3-none-any.whl 462.0 kB Sep 24, 2026	Python 3	none	any	Details

Total release size: 3.3 MB

Release files / genai_otel_instrument-1.30.0.tar.gz

Download URL	genai_otel_instrument-1.30.0.tar.gz
Size	2.8 MB
Tags	Source
SHA-256 checksum How to use checksums	`624deea0c35405162c738033b564b48f301c27b027df606b1f1e1a92a23fda6a`
BLAKE2b-256 checksum How to use checksums	`84ac7204be29a2d83ac3bd343b3894d83f52987525696a600c55585462db366c`
Upload date	Sep 24, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.11.16`

Release files / genai_otel_instrument-1.30.0-py3-none-any.whl

Download URL	genai_otel_instrument-1.30.0-py3-none-any.whl
Size	462.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`19c6e8642f5409d47096bd24e6962f954723347a99e7d243417d8443b67f317e`
BLAKE2b-256 checksum How to use checksums	`d6f66b5cf410594035378eabb8c095412a018e2c2630b26cf815c167ba21604e`
Upload date	Sep 24, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.11.16`