LLMTrack

⭐ If this saves you money, consider starring the repo — it helps others find it.

Feature-level cost attribution and budget monitoring for Large Language Model (LLM) applications.

The Problem

Your OpenAI dashboard shows this:

Total spend this month: $4,230
Total tokens used: 42M

It does not show this:

document_summarization:   $1,800  (42%)  ← Is this worth it?
customer_support:         $1,200  (28%)  ← Can we use a cheaper model?
search_autocomplete:        $800  (19%)  ← Why did this spike 40% this week?
invoice_extraction:         $430  (11%)  ← This is mission-critical, keep it

llmtrack gives you the second view. Free. Open source. Two lines of code.

Overview

Most AI engineering teams know their total monthly LLM bill. Nobody knows which product feature is causing it.

llmtrack is a lightweight, open-source Python library that wraps your LLM calls with feature tags, computes exact token costs against real-time provider pricing, and produces terminal reports and HTML dashboards showing exactly where your AI budget is going — broken down by feature, model, and time period.

Two lines of code to get started. No external database. No cloud dependency. Fully local.

Demo

Terminal Report

HTML Dashboard

Note: Run python examples/quickstart.py to generate sample data and see the terminal report live.

Key Capabilities

• Feature-Level Attribution: Tag model invocations using context managers or direct API parameters.
• Zero-Friction Auto-Patching: Transparently captures token usage and latency from OpenAI and Anthropic SDKs with fallback protections.
• Thread and Task Isolation: Safe for high-concurrency web servers using thread-local context separation.
• Comprehensive Pricing Engine: Pre-configured per-million token rates for 50+ frontier and open models with fuzzy alias resolution.
• Embedded Persistence: Lightweight SQLite storage with indexing and automatic connection recycling; zero mandatory external database dependencies.
• Actionable Reporting: Rich terminal breakdown tables and self-contained interactive HTML dashboards.
• Budget Alerts: Real-time spending thresholds with support for custom webhook and notification callbacks.
• CLI Utility: Query spend statistics and manage tracking databases directly from the command line.

Installation

Install the base package via pip:

pip install llm-cost-track

To include optional LiteLLM integration:

pip install "llm-cost-track[litellm]"

Quickstart

1. Auto-Patching with OpenAI

CostTracker automatically intercepts calls made through official SDKs when initialized with auto_patch=True (default):

import openai
from llmtrack import CostTracker

tracker = CostTracker()

with tracker.feature("document_summary"):
    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Summarize the quarterly financials."}]
    )

# Print a rich breakdown table to the terminal
tracker.report()

2. Auto-Patching with Anthropic

import anthropic
from llmtrack import CostTracker

tracker = CostTracker()
client = anthropic.Anthropic()

with tracker.feature("customer_support_agent"):
    message = client.messages.create(
        model="claude-3-7-sonnet",
        max_tokens=1024,
        messages=[{"role": "user", "content": "How do I update my billing email?"}]
    )

tracker.report()

3. Global Context Tagging

For modular applications, import the top-level feature context manager directly:

from llmtrack import feature
import openai

with feature("invoice_extraction"):
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Extract line items from invoice #4102."}]
    )

4. Explicit Call Logging

For custom API clients, proxy gateways, or self-hosted model deployments, log calls directly:

from llmtrack import CostTracker

tracker = CostTracker(auto_patch=False)

tracker.log_call(
    model="deepseek-r1",
    input_tokens=1250,
    output_tokens=680,
    feature="code_review",
    latency_ms=840.2,
    metadata={"repository": "backend-core", "pull_request_id": 142}
)

Framework Integration

FastAPI

Use context managers inside route handlers or dependency injection pipelines:

from fastapi import FastAPI, Depends
from pydantic import BaseModel
import openai
from llmtrack import CostTracker

app = FastAPI(title="AI Service")
tracker = CostTracker(db_path="production_metrics.db")

class QueryRequest(BaseModel):
    query: str

@app.post("/api/v1/search")
async def semantic_search(request: QueryRequest):
    with tracker.feature("semantic_search"):
        response = openai.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": request.query}]
        )
    return {"result": response.choices[0].message.content}

@app.get("/api/v1/metrics/costs")
async def get_costs(days: int = 7):
    return tracker.summary(days=days)

Flask

from flask import Flask, request, jsonify
import openai
from llmtrack import CostTracker

app = Flask(__name__)
tracker = CostTracker(db_path="flask_costs.db")

@app.route("/api/summarize", methods=["POST"])
def summarize():
    data = request.get_json() or {}
    content = data.get("text", "")

    with tracker.feature("document_summary"):
        response = openai.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": content}]
        )

    return jsonify({"summary": response.choices[0].message.content})

Storage Backends

SQLite (Default)

Persists events across process restarts in an optimized, indexed SQLite database file:

from llmtrack import CostTracker, SQLiteStorage

storage = SQLiteStorage(db_path="data/attribution.db")
tracker = CostTracker(storage=storage)

In-Memory Storage

Thread-safe, ephemeral storage designed for unit tests, transient jobs, or serverless environments:

from llmtrack import CostTracker, MemoryStorage

tracker = CostTracker(storage=MemoryStorage(), auto_patch=False)

Budget Alerts and Notifications

Register spending thresholds per feature to enforce unit economics and guard against runaway loops:

from llmtrack import CostTracker

tracker = CostTracker()

def on_budget_exceeded(feature_name: str, spent_usd: float, limit_usd: float):
    # Dispatch alert via Slack, PagerDuty, or email
    print(f"CRITICAL: Feature '{feature_name}' reached ${spent_usd:.2f} (limit: ${limit_usd:.2f})")

tracker.set_budget_alert(
    feature="bulk_document_indexing",
    daily_limit_usd=25.00,
    callback=on_budget_exceeded
)

Reporting

Terminal Output

Generate formatted Rich summary tables directly in stdout:

tracker.report(days=7, output="terminal")

Sample output:

-------------------------------------------------------------------------------------
Feature                   Cost (USD)     % of Total   Calls      Avg/Call    Tokens
-------------------------------------------------------------------------------------
document_summary             $12.4500         58.2%     120     $0.103750   1,420,000
customer_support              $6.8200         31.9%     450     $0.015155     890,000
semantic_search               $2.1200          9.9%     840     $0.002523     310,000
-------------------------------------------------------------------------------------
Total Spend: $21.3900 | Total Calls: 1,410 | Period: Last 7 Days
-------------------------------------------------------------------------------------

Standalone HTML Dashboards

Export interactive, dark-mode dashboards with metrics cards, percentage breakdown visualizers, and sortable data grids:

tracker.report(output="html", filepath="reports/weekly_cost_report.html", days=7)

Programmatic Aggregations

Retrieve raw aggregations as Python dictionaries:

summary = tracker.summary(days=30)
total_cost = summary["total_cost_usd"]
feature_stats = summary["features"]

Supported Models

The built-in pricing engine includes updated rates (per 1,000,000 tokens) and longest-match fuzzy aliasing for the following model families:

Custom Model Registration

Register proprietary or newly released models dynamically at runtime:

from llmtrack.pricing.updater import register_custom_model

register_custom_model(
    model="internal-fine-tuned-llama-70b",
    input_price_per_1m=0.35,
    output_price_per_1m=0.75,
    aliases=["custom-llama-v1", "internal-model"]
)

Command Line Interface

llmtrack includes a CLI utility for interacting with persistent storage files:

# Display summary table for the last 7 days (default database: llmtrack.db)
llmtrack report

# Display summary for the last 30 days against a custom database path
llmtrack report --days 30 --db /var/data/llmtrack.db

# Export an HTML dashboard
llmtrack report --html --output reports/monthly_spend.html --days 30

# Clear recorded telemetry from database
llmtrack clear --db /var/data/llmtrack.db

API Reference

CostTracker

CostTracker(
    storage: Optional[BaseStorage] = None,
    db_path: str = "llmtrack.db",
    auto_patch: bool = True
)

• feature(name: str) -> Generator[None, None, None]
  Thread-safe context manager that associates all underlying LLM calls with name. Supports nesting.
• log_call(model: str, input_tokens: int, output_tokens: int, feature: Optional[str] = None, latency_ms: float = 0.0, metadata: Optional[dict] = None) -> CallEvent
  Records an individual model call event and persists it to configured storage.
• report(days: int = 7, output: str = "terminal", filepath: Optional[str] = None) -> None
  Renders formatted terminal statistics or writes an interactive HTML dashboard to disk.
• summary(days: int = 7) -> dict
  Returns aggregated cost, token, and call count metrics grouped by feature.
• set_budget_alert(feature: str, daily_limit_usd: float, callback: Optional[Callable] = None) -> None
  Establishes a daily spending threshold for a designated feature.

Real-World Impact

Teams using feature-level cost attribution typically find:

Bottom line: Most teams find 20–40% cost reduction opportunities within the first week of tracking.

Contributing

Contributions, issue reports, and pull requests are welcome.

1. Fork the repository on GitHub.
2. Create a feature branch: git checkout -b feature/model-expansion
3. Ensure test suite passes: pytest --cov=llmtrack
4. Commit your changes: git commit -m 'Add support for new provider'
5. Push to your branch and open a Pull Request at https://github.com/ManikBodamwad/LLMTracker.

License

This project is licensed under the MIT License. See the LICENSE file for full terms.