TraceVerde
The most comprehensive OpenTelemetry auto-instrumentation library for LLM/GenAI applications
Trace from OpenTelemetry traces. Verde meaning green - for sustainable, transparent AI observability.
Documentation | Examples | Discord | PyPI
Get Started in 30 Seconds
pip install genai-otel-instrument
import genai_otel
genai_otel.instrument()
# Your existing code works unchanged - traces, metrics, and costs are captured automatically
import openai
client = openai.OpenAI()
response = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "Hello!"}])
That's it. No wrappers, no decorators, no config files. Every LLM call, database query, and agent interaction is automatically traced with full cost breakdown.
Let your coding agent do it
Using an AI coding agent or assistant? Paste this prompt into it, in the repository you want instrumented. It adds the two lines above in the right place, installs the right extras, configures through environment variables, and verifies the result with a real run. The same prompt is in the docs.
Copy the prompt
Add OpenTelemetry tracing for LLM calls to this Python application using TraceVerde.
Package on PyPI: genai-otel-instrument Import name: genai_otel
Docs: https://mandark-droid.github.io/genai_otel_instrument/
Steps
1. Find the entry point of every process this application starts: the module that
runs first (CLI main, ASGI/WSGI app module, worker start-up script, the first cell
of a notebook).
2. Find which LLM SDKs and agent frameworks the code imports, for example openai,
anthropic, langchain, langgraph, crewai, pydantic_ai, autogen_agentchat. Install
the package with the matching extras and add it to the project's dependency file
(requirements.txt, pyproject.toml, or whatever this project uses):
pip install "genai-otel-instrument[openai,langgraph]"
Extras are named after the library: openai, anthropic, google, langchain,
langgraph, crewai, pydantic-ai, autogen-agentchat, llamaindex, and others listed
in the docs. Plain "pip install genai-otel-instrument" is enough when the SDK is
already a dependency of this project.
3. At the very top of each entry point, add:
import genai_otel
genai_otel.instrument()
This must run before any LLM SDK or agent framework is imported, and before any
client, LLM or agent object is created. A client created earlier is not traced.
Call it once per process. Do not wrap, decorate or otherwise change existing LLM
calls: instrumentation is automatic.
4. Configure through environment variables, not code. Add them wherever this project
keeps its configuration (.env.example, deployment manifests, the README):
OTEL_SERVICE_NAME=<a name for this service>
OTEL_EXPORTER_OTLP_ENDPOINT=<OTLP/HTTP endpoint, for example http://localhost:4318>
OTEL_EXPORTER_OTLP_HEADERS=<key1=val1,key2=val2, only if the backend needs auth>
Never hard-code endpoints, API keys or headers. Leave content capture off, which
is the default: GENAI_ENABLE_CONTENT_CAPTURE=true records prompts and completions
on spans, so only turn it on if I ask for it.
5. Only if I ask for it:
- limit what is instrumented: GENAI_ENABLED_INSTRUMENTORS=openai,langgraph
- sample traces: GENAI_SAMPLING_RATE=0.5
6. Verify with a real run. Run the smallest path of the application that makes an
LLM call, with OTEL_EXPORTER_OTLP_ENDPOINT set to an empty string for that run.
Spans are then printed to the console instead of exported. Check that:
- there is one span per LLM call, carrying gen_ai.request.model,
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and
gen_ai.usage.cost.total when the model has a known price;
- if an agent framework is used, its spans are the parents of the LLM spans, in
the same trace.
Then restore the real endpoint.
7. Report back: the files you changed, the install line, the environment variables to
set, and the span names you saw in step 6. If you could not verify something, say
so instead of assuming it works.
Rules
- Make the smallest change that works. Do not refactor unrelated code.
- Do not add a second LLM tracing library next to this one. If one is already
present, or the application already sets up its own OpenTelemetry TracerProvider,
stop and ask me how to proceed.
- If an agent framework ships its own tracing, leave it off unless I ask; running
both reports every model request twice.
- Do not commit secrets, and do not print or log API keys.
Why TraceVerde?
| Feature | TraceVerde | OpenLIT | Traceloop/OpenLLMetry | Langfuse | Galileo | Arize (Phoenix) | Opik (Comet) |
|---|---|---|---|---|---|---|---|
| Zero-code setup | Yes | Yes | Yes | SDK required | SDK required | SDK / auto (OpenInference) | SDK required |
| LLM providers | 26 | 25+ | 15+ | Via integrations | Via integrations | Via integrations | Via integrations |
| Multi-agent frameworks | 9 (CrewAI, LangGraph, ADK, Antigravity, AutoGen, OpenAI Agents, Pydantic AI, etc.) | Limited | Limited | Limited | Limited | Limited | Limited |
| Cost tracking | Automatic (1,700+ models) | Manual config | Manual config | Manual config | Yes | Yes | Yes |
| GPU metrics (NVIDIA + AMD) | Yes | No | No | No | No | No | No |
| MCP tool instrumentation | Yes (databases, caches, vector DBs, queues) | Limited | Limited | No | No | No | No |
| Evaluation (PII, toxicity, bias, hallucination, prompt injection) | Built-in (6 detectors) | No | No | Separate service | Extensive (core focus) | Built-in (Phoenix evals) | Built-in (core focus) |
| OpenTelemetry native | Yes | Yes | Yes | Partial | Partial | Yes (OpenInference) | Partial |
| Self-hosted / on-prem | Yes (fully local) | Yes | Yes | Yes | Enterprise tier | Yes (Phoenix) | Yes |
| License | Apache-2.0 | Apache-2.0 | Apache-2.0 | MIT | Proprietary | Open-source + Commercial | Apache-2.0 |
What Gets Instrumented?
LLM Providers (26+)
OpenAI, OpenRouter, CometAPI, Anthropic, Google AI, Google GenAI, AWS Bedrock, Azure OpenAI, Cohere, Mistral AI, Together AI, Groq, Ollama, vLLM, llama.cpp, Vertex AI, Replicate, HuggingFace, SambaNova, Sarvam AI, ElevenLabs, Hyperbolic, LiteLLM, TypeSafe AI / Jev
See all providers with examples >>
Multi-Agent Frameworks (8)
CrewAI, LangGraph, Google ADK, AutoGen, AutoGen AgentChat, OpenAI Agents SDK, Pydantic AI, AWS Bedrock Agents
See all frameworks with examples >>
MCP Tools (20+)
Databases: PostgreSQL, MySQL, MongoDB, SQLAlchemy, TimescaleDB, OpenSearch, Elasticsearch, FalkorDB Caching: Redis | Queues: Kafka, RabbitMQ | Storage: MinIO Vector DBs: Pinecone, Weaviate, Qdrant, ChromaDB, Milvus, FAISS, LanceDB
Built-in Evaluation (6 Detectors)
PII Detection (GDPR/HIPAA/PCI-DSS), Toxicity Detection, Bias Detection, Prompt Injection Detection, Restricted Topics, Hallucination Detection
See all evaluation features with examples >>
Screenshots
OpenAI traces with token usage, costs, and latency
Key Features
Automatic Cost Tracking
1,700+ models across 40+ providers with per-request cost breakdown. Supports differential pricing (prompt vs completion), reasoning tokens, cache pricing, and custom model pricing.
# Cost tracking is enabled by default - just instrument and go
genai_otel.instrument()
# Or add custom pricing for proprietary models
export GENAI_CUSTOM_PRICING_JSON='{"chat":{"my-model":{"promptPrice":0.001,"completionPrice":0.002}}}'
GPU Metrics (NVIDIA + AMD)
Real-time monitoring of utilization, memory, temperature, power, PCIe throughput, throttling, and ECC errors. Multi-GPU aggregate metrics included.
pip install genai-otel-instrument[gpu] # NVIDIA
pip install genai-otel-instrument[amd-gpu] # AMD
Multi-Agent Tracing
Complete span hierarchy for agent frameworks with automatic context propagation:
Crew Execution
+-- Agent: Senior Researcher (gpt-4)
| +-- Task: Research OpenTelemetry
| +-- openai.chat.completions (tokens: 1250, cost: $0.03)
+-- Agent: Technical Writer (ollama:llama2)
+-- Task: Write blog post
+-- ollama.chat (tokens: 890, cost: $0.00)
Multimodal Observability (v1.1.0)
First-class capture of image, audio, video, and document content parts on OpenAI, Anthropic, Google Gemini, and Groq spans, and of the audio on Sarvam and ElevenLabs speech calls (the audio sent for transcription, the audio text-to-speech returns; the transcript and spoken text follow content capture). Bytes are offloaded to your configured object store (MinIO / S3 / filesystem / HTTP) and referenced from spans by URI — they never appear inline in span attributes.
# Opt in (default is off — text-only behaviour is byte-identical to 1.0.x)
export GENAI_OTEL_MEDIA_CAPTURE_MODE=full
export GENAI_OTEL_MEDIA_STORE=minio
export GENAI_OTEL_MEDIA_STORE_ENDPOINT=http://localhost:9000
export GENAI_OTEL_MEDIA_STORE_ACCESS_KEY=...
export GENAI_OTEL_MEDIA_STORE_SECRET_KEY=...
# Optional: plug in a redactor before upload
export GENAI_OTEL_MEDIA_REDACTOR=genai_otel.media.redactors.face_blur
Spans get two co-emitted representations of the same multimodal content:
- A flat, queryable attribute namespace —
gen_ai.prompt.{n}.content.{m}.{type, media_uri, media_mime_type, media_byte_size, media_source}plus agen_ai.completion.*mirror — for backends that index on flat attributes. - The upstream-canonical
gen_ai.input.messages/gen_ai.output.messagesJSON conforming to the gen-ai message schemas in the dedicatedsemantic-conventions-genairepo, including thedocumentmodality standardised by our upstream PR #142, plus the optionalbyte_sizeandstripped_reasonshapes proposed in #143 / #144 (see Standards Contributions below).
Safety & Evaluation
genai_otel.instrument(
enable_pii_detection=True, # GDPR/HIPAA/PCI-DSS compliance
enable_toxicity_detection=True, # Perspective API + Detoxify
enable_bias_detection=True, # 8 bias categories
enable_prompt_injection_detection=True,
enable_hallucination_detection=True,
enable_restricted_topics=True,
)
Evaluation guide with 50+ examples >>
Configuration
# Required
export OTEL_SERVICE_NAME=my-llm-app
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# Optional
export GENAI_ENABLE_GPU_METRICS=true
export GENAI_ENABLE_COST_TRACKING=true
export GENAI_SAMPLING_RATE=0.5 # Reduce volume in production
export GENAI_ENABLED_INSTRUMENTORS=openai,crewai # Select specific instrumentors
Full configuration reference >>
Backend Integration
Works with any OpenTelemetry-compatible backend:
Jaeger, Zipkin, Prometheus, Grafana, Datadog, New Relic, Honeycomb, AWS X-Ray, Google Cloud Trace, Elastic APM, Splunk, SigNoz, self-hosted OTel Collector
Pre-built Grafana dashboard templates included.
Examples
90+ ready-to-run examples covering every provider, framework, and evaluation feature:
examples/
+-- openai/ # OpenAI chat, embeddings
+-- anthropic/ # Anthropic + PII/toxicity detection
+-- ollama/ # Local models + all evaluation features
+-- crewai_example.py # Multi-agent crew orchestration
+-- langgraph_example.py # Stateful graph workflows
+-- google_adk_example.py # Google Agent Development Kit
+-- antigravity/ # Google Antigravity SDK agent turns
+-- autogen_example.py # Microsoft AutoGen agents
+-- pii_detection/ # 10 PII examples (GDPR, HIPAA, PCI-DSS)
+-- toxicity_detection/ # 8 toxicity examples
+-- bias_detection/ # 8 bias examples (hiring compliance, etc.)
+-- prompt_injection/ # 6 injection defense examples
+-- hallucination/ # 4 hallucination detection examples
+-- ... # And many more
OpenTelemetry Standards Contributions
TraceVerde isn't only a consumer of OpenTelemetry GenAI semantic conventions — production gaps surfaced by the library are being upstreamed back into the spec, and one is now part of the standard. Proposals on open-telemetry/semantic-conventions-genai:
| PR | Proposal | Status |
|---|---|---|
| #142 | Add document to the Modality enum on BlobPart / FilePart / UriPart — PDFs, DOCX, and other non-image/video/audio payloads currently fall through to the free-form string branch. BFSI KYC extraction is a high-volume real example. |
Merged 2026-05-16 |
| #143 | Add optional byte_size on the three media-part types so consumers get a uniform handle on payload size whether the content was carried inline, by URI, or by provider file id — useful for cost-of-capture telemetry and storage planning. Pydantic ge=0 → JSON schema "minimum": 0. |
Open — reference scenarios for FilePart / UriPart requested |
| #144 | Make content / file_id / uri optional and add a free-form stripped_reason (size_exceeded, modality_not_allowed, redactor_error, upload_error, no_store_configured) so an instrumentation can fail-closed — record that it observed a media part but intentionally did not capture its bytes — while preserving the original part type and modality. Enforced via a top-level anyOf so structurally-empty parts cannot validate. |
Open — with reviewers |
All three were migrated from the closed open-telemetry/semantic-conventions#3673 after the GenAI conventions split into the dedicated repo on 2026-05-05. Each PR ships under the new repo's V2 Weaver schema with the corresponding models.ipynb updates and make check-policies / make generate-all validation.
TraceVerde already emits this shape on the wire, providing the reference implementation for these conventions. Since v1.10.0 the GenAI semconv tier is opted into by default (OTEL_SEMCONV_STABILITY_OPT_IN defaults to gen_ai/dup), so spans carry both the current and the superseded attribute spellings.
Ecosystem & Framework Contributions
Beyond the spec, genai_otel is shipping inside agent frameworks as their OpenTelemetry observability layer — in-tree where the project accepts it, or as a standalone plugin where the project's policy keeps integrations out of the core tree — demonstrating the library powering real third-party agents, not just first-party services:
| Integration | Framework | Contribution | Status |
|---|---|---|---|
| hermes-otel-plugin | NousResearch Hermes | A standalone otel plugin (hermes plugins install Mandark-droid/hermes-otel-plugin) exporting Hermes turns / LLM calls / tool calls as OTel GenAI spans and dashboard log records, with no changes to Hermes core. When genai-otel-instrument is installed it additionally unlocks on-prem GPU / energy / CO2 metrics, local-model cost (parameter-size pricing for Ollama / HF / vLLM), and an inline eval / guardrail suite (PII, toxicity, bias, prompt-injection, restricted-topics, hallucination) scored on prompt and response — signals no vanilla OTel SDK or other GenAI instrumentor emits. Originally upstream PR #48184 — approved on code review, then republished standalone per Hermes's policy that observability backends ship as standalone plugin repos. |
Shipped (standalone plugin, MIT) |
Featured In
- Context Engineering - AI ecosystem - TraceVerde is listed under Assurance & governance > Observability in the AI ecosystem catalog that accompanies Boni García's book Context Engineering: Build Consistent, Accurate, Predictable AI Systems (Manning).
Who Uses TraceVerde?
TraceVerde is used by developers and teams building production GenAI applications. If you're using TraceVerde, we'd love to hear from you!
Add your company | Join Discord
Community
- Documentation - Full guides, API reference, and tutorials
- Discord - Chat with the community
- GitHub Issues - Bug reports and feature requests
- Contributing - How to contribute
- Human and LLM agent contributions - How agents can report gaps or submit focused fixes
License
Copyright 2025 Kshitij Thakkar. Licensed under the Apache License 2.0.
1 maintainer
Metadata
1 maintainer
Release files for genai-otel-instrument 1.36.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| genai_otel_instrument-1.36.1.tar.gz | 3.0 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| genai_otel_instrument-1.36.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 3.6 MB
Release files / genai_otel_instrument-1.36.1.tar.gz
| Download URL | genai_otel_instrument-1.36.1.tar.gz |
|---|---|
| Size | 3.0 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a88bbab2e91fc6be4e651df91dc821a75e7ec894a28849f98c1bb578c891a184
|
|
BLAKE2b-256 checksum How to use checksums |
bea1da3f61b6259f4babbf225b7fe7d60c9fdeba3bad16ec957784da67892ec8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.17
|
Release files / genai_otel_instrument-1.36.1-py3-none-any.whl
| Download URL | genai_otel_instrument-1.36.1-py3-none-any.whl |
|---|---|
| Size | 542.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
29b63bb1039255eb802d039b5309edf54bb85e2c05fb5f1bb73faf11ff1a75eb
|
|
BLAKE2b-256 checksum How to use checksums |
09518039c430a15fb412f1910ca79575b6a3ce7b28d2f03b49749a19f25783a7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.17
|
