costopt 0.1.2

Drop-in LLM API cost optimization SDK & local developer observability platform.
Stop paying for redundant LLM calls. Intercept, route, cache, and optimize prompt spend before requests hit paid APIs.

# CostOpt — Cost-Aware Local Development & Feature-Level LLM Cost Attribution ⚡

Don't wait for a $500 monthly cloud bill to figure out where your LLM budget went. CostOpt puts real-time cost attribution directly into your local development loop—giving you feature-level spend visibility, lexical response caching, and automated optimization recommendations before code ships.

💡 Why CostOpt?

Most LLM gateways operate in production infrastructure after code is shipped. CostOpt operates in your local development loop:

1. Feature-Level Cost Attribution: Group LLM calls by feature or component (feature="rag_summarizer") to track feature unit economics before shipping.
2. Local Lexical Cache: High-speed token & n-gram similarity cache returning sub-2ms latency and $0.00 cost on repeated prompts.
3. Silent Loop Circuit Breaker: Detects rapid call loops (>15 calls in 30s) and trips CostOptCircuitBreakerError locally to stop runaway billing leaks before they happen.
4. Zero-Downtime Outage Failover: Automatically reroutes queries to multi-provider fallbacks (gpt-4o → claude-3-5-sonnet or local ollama/llama3) when primary providers return 429 Rate Limits or 503 Outages.
5. Zero-Churn 1-Line SDK Interception: Patches standard OpenAI client calls with zero architectural refactoring.

🖥️ Developer Observability Console

Live System Overview displaying spend metrics, vector cache hits, optimization recommendations, and prompt interception logs.

Dedicated Trace Explorer auditing prompt MD5 hashes, response latencies, model rerouting decisions, and status code badges.

🏗️ Architecture & Request Flow

graph TD
    App["💻 Application Code"] -->|client.chat.completions.create| Interceptor["⚡ CostOpt Middleware"]
    
    Interceptor -->|1. Vector Cosine Lookup| Cache{"💾 SQLite Vector Cache"}
    Cache -->|Cache HIT 0ms / $0.0| App
    
    Cache -->|Cache MISS| Router{"🧠 Complexity Router"}
    Router -->|Simple Query| MiniModel["🚀 Mini / Local Ollama ($0.0)"]
    Router -->|Complex Query| OriginalModel["🌐 Cloud Provider API ($$$)"]
    
    MiniModel --> Telemetry["📊 Local SQLite Telemetry Logger"]
    OriginalModel --> Telemetry
    Telemetry --> Dashboard["🖥️ Local Observability Dashboard (Port 8000)"]

🚀 Quickstart

1. Installation

pip install costopt

2. Basic Integration

from openai import OpenAI
from costopt import CostOpt

# Wrap standard client in one line
client = CostOpt(OpenAI(api_key="your-api-key"))

# Requests are automatically intercepted, cached, and optimized!
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Classify sentiment: I love python!"}]
)

3. Launch Observability Dashboard

costopt dashboard

Open http://localhost:8000 in your browser to view real-time spend analytics, trace logs, and policy rules!

4. Integration with Popular Frameworks

CostOpt wraps standard OpenAI-compatible client instances in 1 line:

LangChain:

from langchain_openai import ChatOpenAI
from costopt import CostOpt

# Wrap underlying client
llm = ChatOpenAI(client=CostOpt(OpenAI()).client)

LlamaIndex:

from llama_index.llms.openai import OpenAI as LlamaOpenAI
from costopt import CostOpt

llm = LlamaOpenAI(client=CostOpt(OpenAI()).client)

FastAPI Middleware Integration:

from fastapi import FastAPI
from openai import OpenAI
from costopt import CostOpt

app = FastAPI()
ai_client = CostOpt(OpenAI())

🔌 VS Code Extension — Cost Intelligence While You Code

CostOpt includes a native VS Code Extension that brings real-time LLM cost intelligence directly into your editor:

1. Install CostOpt & VS Code Extension:

   pip install costopt
   

   Install CostOpt from the VS Code Extensions Marketplace or load vscode-extension/.

2. Start Local Service:

   costopt dashboard
   

3. Make LLM Calls: As you call your LLM code locally or run test suites, CostOpt displays:

   • Live CodeLens: Line-level cost (CostOpt: ~$0.012 / request | Avg tokens: 3,421 | Calls: 184) directly above your code.
   • Hover Panels: Rich, compact cost breakdowns when hovering over client.chat.completions.create().
   • Status Bar Item: Compact daily spend tracker (CostOpt: $8.42 today).
   • Feature Attribution Sidebar: Breakdown of spend grouped by feature (@costopt.track("customer_support") or feature="summarizer").
   • Cost Forecast & Warnings: Spend forecasting and cost drift warnings directly in the Problems panel.

🔧 Configuration Guide

Custom Models & User Local Overrides

Track custom, fine-tuned, or local models by dropping a .yaml file into your project:

provider: "ollama"
models:
  deepseek-r1:
    input_cost_per_1m: 0.0
    output_cost_per_1m: 0.0

Pass the pricing directory:

client = CostOpt(OpenAI(), pricing_dir="./my_pricing")

🛡️ Security Audit

CostOpt has undergone automated penetration testing for SQL injections, CORS misconfigurations, and rate-limiting DB locks. See the full audit report at docs/SECURITY_AUDIT.md.

📄 License

This project is licensed under the MIT License. See LICENSE for details.