datametry 0.1.1a0

LLM observability SDK for OpenAI and Anthropic

Datametry SDK

Observability and tracing for OpenAI & Anthropic workloads – built for modern LLM Ops.

datametry is a zero-code observability layer for LLM applications. It hooks into the official OpenAI and Anthropic SDKs, captures detailed telemetry (latency, token usage, params, errors), enriches it with OpenTelemetry context, and streams everything to the Datametry backend via a resilient exporter.

---

Capability Highlights (v0.1.0-alpha)

• Automatic provider instrumentation – monkey patches OpenAI chat.completions.create and Anthropic messages.create with one call to init_observability().
• Rich span payloads – request/response payloads, finish reasons, structured token usage (real usage when available, otherwise tiktoken-powered estimates), error objects with stack traces, and trace correlations.
• Custom tracing hooks – decorators, context managers, and helpers (observe, trace_context, add_span_attribute, add_span_event) powered by OpenTelemetry for arbitrary Python functions.
• Production-focused exporter – lock-free queue, batching, retry/backoff, graceful shutdown, and queue backpressure so your app stays responsive.
• OpenTelemetry bridge – automatically starts llm.request spans if none exist, or enriches your active spans with token counts and finish reasons.

---

Installation

# Poetry
poetry add datametry

# Pip
pip install datametry

Requires Python 3.13+ and an active Datametry API key.

---

Quick Start

1. Export credentials

export DATAMETRY_API_KEY="sk_test_..."   # required
export DATAMETRY_ENABLED=true              # optional (default true)

2. Instrument your app

from datametry import init_observability, observe
import openai, anthropic

init_observability()

openai_client = openai.OpenAI()
anthropic_client = anthropic.Anthropic()

# OpenAI – fully automatic tracking
response = openai_client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize SOC2."}],
    temperature=0.2,
)

# Anthropic – same queue, no extra work
claude = anthropic_client.messages.create(
    model="claude-3-5-haiku-latest",
    max_tokens=200,
    messages=[{"role": "user", "content": "Summarize SOC2."}],
)

# Custom business logic
@observe(name="process_ticket", attributes={"tier": "gold"})
def process_ticket(ticket_id: str, body: str) -> str:
    ...
    return "ok"

3. Optional explicit shutdown

from datametry import shutdown_observability

shutdown_observability(timeout=10)  # flush remaining spans

---

Configuration Surface

All knobs are environment variables (overrideable via init_observability()):

Variable	Purpose	Default
DATAMETRY_API_KEY	Auth token for ingest	required
DATAMETRY_API_URL	Ingest endpoint	https://api.datametry.co/v1/ingest
DATAMETRY_ENABLED	Master switch	true
DATAMETRY_BATCH_SIZE	Export batch size	100
DATAMETRY_FLUSH_INTERVAL	Seconds between flushes	5.0
DATAMETRY_QUEUE_MAX_SIZE	In-memory queue bound	10000
DATAMETRY_MAX_RETRIES	Export retry attempts	3
DATAMETRY_REQUEST_TIMEOUT	HTTP timeout (s)	10.0

Programmatic overrides:

init_observability(
    api_key="sk_test...",
    enabled=True,
    log_level="DEBUG",
    enable_tracing=True,
)

---

Telemetry Model

Every captured span is decomposed into three payload types and queued together:

• LLM Requests – provider, model, timestamp, latency, finish_reason, full prompt/messages, captured outputs, request params, OpenTelemetry identifiers.
• Token Usage – prompt/completion/total token counts; uses real values when provided by the SDK or falls back to tiktoken or character-based heuristics.
• Errors – Python exception type/code/message plus truncated stack traces.
• Trace Spans – when observe/trace_context are used, full OTel spans (attributes, events, IO metadata) are exported alongside LLM calls for end-to-end correlation.

---

Architecture Snapshot

Your App (OpenAI / Anthropic / custom)
        ↓ monkey-patched clients & decorators
Datametry Instrumentation (LLM + OTEL)
        ↓
Thread-safe queue (backpressure & drop-old)
        ↓
Background exporter thread
        ↓ async HTTP client (httpx)
Datametry ingest API (batch + retry)

Key guarantees:

• Non-blocking: SDK calls enqueue telemetry without touching the network path.
• Resilient exporter: time or size-based flush, exponential backoff, graceful shutdown hook via atexit.
• Queue safety: bounded queue protects your app; oldest spans are dropped if pressure persists.

---

Instrumentation Coverage

Provider	API	Status	Notes
OpenAI	chat.completions.create (sync)	✅	Token usage extracted when OpenAI returns usage; otherwise estimated.
OpenAI	Streaming (stream=True)	⚠️ Partial	See examples/test_streaming.py for current behavior; chunk-level telemetry still WIP.
Anthropic	messages.create (sync)	✅	Handles system prompt + tool use blocks; always captures usage.
Anthropic	Streaming	⚠️ Partial	Events enumerated in tests; telemetry enrichment planned.
Custom code	@observe, trace_context	✅	OpenTelemetry spans exported via same queue.

Roadmap items (tracked via GitHub issues): async client support, streaming deltas, additional providers, custom span ingestion, SDK-specific extras.

---

Examples

The examples/ directory doubles as runnable docs:

• basic_usage.py – OpenAI quick start and error capture.
• advanced_usage.py – custom configuration, workload simulation, batching demo.
• multi-llm.py – single init that tracks both OpenAI and Anthropic.
• example_tracing.py – custom tracing helpers (functions, contexts, events).
• test_streaming.py – current streaming behavior + gap analysis.

Run with Poetry:

poetry install
poetry run python examples/basic_usage.py

---

Troubleshooting Cheatsheet

• Missing API key – set DATAMETRY_API_KEY or pass api_key= to init_observability.
• OpenAI / Anthropic not found – install the relevant SDK (pip install openai anthropic); instrumentation gracefully skips absent providers.
• tiktoken warnings – install tiktoken for precise usage numbers; otherwise we fallback to character estimates.
• Need verbose logs – pass log_level="DEBUG" to surface exporter + instrumentation details.
• Lingering spans on exit – call shutdown_observability(timeout=10) before terminating short-lived scripts/tests.

---

Contributing & Local Dev

git clone https://github.com/datametry-official/datametry-sdk
cd datametry
poetry install
poetry run pytest       # coming soon
poetry run python examples/basic_usage.py

We welcome PRs for new providers, better streaming support, and tracing improvements. Please discuss major changes via GitHub issues first.

---

License & Links

• License: MIT (see LICENSE)
• Website: datametry.co
• Docs: docs.datametry.co

Made with ❤️ by the Datametry team.