LLM Usage Framework

A generic, provider-independent Python framework for tracking LLM usage and collecting standardized telemetry.

Features

• Multiple LLM provider support
• Standardized LLMTracker interface
• Input, output, and total token tracking
• Cache usage tracking when reported by the provider
• Token source tracking
• Character count tracking
• Latency tracking
• Provider and model tracking
• Project, service, feature, and metadata support
• Request ID and session ID tracking
• Optional tokenizer fallback
• Optional on_usage callback
• LLMConfig support
• Provider-specific parameters through provider_kwargs
• Storage-agnostic
• No database dependency

Supported Providers

Installation

Install the core package:

pip install llm-usage-framework

Usage

Basic Example (Ollama)

Here is a simple example of how to use the LLMTracker with the Ollama provider:

from llm_usage.tracker import LLMTracker

# Initialize the tracker
tracker = LLMTracker(
    provider="ollama",
    model="qwen3:8b",
    project="my_project",
    service="my_service",
    provider_kwargs={
        "base_url": "http://localhost:11434",
    },
)

# Generate a response
result = tracker.generate(
    prompt="Say hello in one sentence.",
)

# The result contains standardized usage metrics
print(result.text)
print(f"Input tokens: {result.usage.input_tokens}")
print(f"Output tokens: {result.usage.output_tokens}")
print(f"Total tokens: {result.usage.total_tokens}")
print(f"Latency: {result.latency_ms} ms")

Usage with OpenAI

You can easily switch to a different provider by changing the initialization parameters:

from llm_usage.tracker import LLMTracker
import os

tracker = LLMTracker(
    provider="openai",
    model="gpt-4",
    project="my_project",
    service="my_service",
    provider_kwargs={
        "api_key": os.getenv("OPENAI_API_KEY"),
    },
)

result = tracker.generate(
    prompt="Explain quantum computing in simple terms.",
)

print(result.text)
print(f"Usage: {result.usage.total_tokens} tokens")

Usage with AWS Bedrock

You can also use AWS Bedrock by specifying the model and passing the region_name in the provider kwargs. Ensure you have your AWS credentials configured (e.g., via environment variables or the ~/.aws/credentials file):

from llm_usage.tracker import LLMTracker

tracker = LLMTracker(
    provider="bedrock",
    model="amazon.nova-micro-v1:0",
    project="my_project",
    service="my_service",
    provider_kwargs={
        "region_name": "us-east-1",
    },
)

result = tracker.generate(
    prompt="What is cloud computing?",
)

print(result.text)
print(f"Usage: {result.usage.total_tokens} tokens")

Usage with LLMConfig

You can also encapsulate your configuration using the LLMConfig object. This is especially useful when you need to pass the configuration across different parts of your application:

from llm_usage.tracker import LLMTracker
from llm_usage.config import LLMConfig

# Define the framework configuration
config = LLMConfig(
    provider="anthropic",
    model="claude-3-haiku-20240307",
    project="my_project",
    service="my_service",
    provider_kwargs={
        "api_key": "YOUR_ANTHROPIC_API_KEY",
    },
)

# Initialize the tracker with the config
tracker = LLMTracker(config=config)

result = tracker.generate(
    prompt="Write a short poem about the ocean.",
)

print(result.text)
print(f"Usage: {result.usage.total_tokens} tokens")