costopt 0.1.0

Drop-in LLM API cost optimization SDK & local developer observability platform.
Stop paying for redundant LLM calls. Intercept, route, cache, and optimize prompt spend before requests hit paid APIs.

📌 Key Capabilities

• Zero-Churn 1-Line SDK Interception: Drop-in wrapper patches standard completion clients (client = CostOpt(OpenAI())) with zero modifications to existing calling code.
• Automated Cost Optimization: Intelligent intent detection automatically routes lightweight queries (like sentiment or text formatting) from expensive models (gpt-4o) to low-cost models (gpt-4o-mini or local llama3), preserving response quality while cutting spend.
• Lexical Similarity Cache: High-speed token & character n-gram similarity cache returns sub-2ms latency, $0.00 cost on repeated or similar prompts.
• Local & Offline Model Support: Seamlessly route to local Ollama models (llama3, mistral, deepseek-r1, qwen2.5) for 100% free offline execution.
• 100% Private Local Telemetry: Logs financial metrics, latency distributions, and MD5 trace hashes to a local SQLite database—zero data shared with third-party servers.

🖥️ Developer Observability Console

Live System Overview displaying spend metrics, vector cache hits, optimization recommendations, and prompt interception logs.

Dedicated Trace Explorer auditing prompt MD5 hashes, response latencies, model rerouting decisions, and status code badges.

🏗️ Architecture & Request Flow

graph TD
    App["💻 Application Code"] -->|client.chat.completions.create| Interceptor["⚡ CostOpt Middleware"]
    
    Interceptor -->|1. Vector Cosine Lookup| Cache{"💾 SQLite Vector Cache"}
    Cache -->|Cache HIT 0ms / $0.0| App
    
    Cache -->|Cache MISS| Router{"🧠 Complexity Router"}
    Router -->|Simple Query| MiniModel["🚀 Mini / Local Ollama ($0.0)"]
    Router -->|Complex Query| OriginalModel["🌐 Cloud Provider API ($$$)"]
    
    MiniModel --> Telemetry["📊 Local SQLite Telemetry Logger"]
    OriginalModel --> Telemetry
    Telemetry --> Dashboard["🖥️ Local Observability Dashboard (Port 8000)"]

🚀 Quickstart

1. Installation

pip install git+https://github.com/khusshdesai/CostOpt.git

PyPI release coming soon — track progress in Issues.

2. Basic Integration

from openai import OpenAI
from costopt import CostOpt

# Wrap standard client in one line
client = CostOpt(OpenAI(api_key="your-api-key"))

# Requests are automatically intercepted, cached, and optimized!
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Classify sentiment: I love python!"}]
)

3. Launch Observability Dashboard

costopt dashboard

Open http://localhost:8000 in your browser to view real-time spend analytics, trace logs, and policy rules!

4. Integration with Popular Frameworks

CostOpt wraps standard OpenAI-compatible client instances in 1 line:

LangChain:

from langchain_openai import ChatOpenAI
from costopt import CostOpt

# Wrap underlying client
llm = ChatOpenAI(client=CostOpt(OpenAI()).client)

LlamaIndex:

from llama_index.llms.openai import OpenAI as LlamaOpenAI
from costopt import CostOpt

llm = LlamaOpenAI(client=CostOpt(OpenAI()).client)

FastAPI Middleware Integration:

from fastapi import FastAPI
from openai import OpenAI
from costopt import CostOpt

app = FastAPI()
ai_client = CostOpt(OpenAI())

🔧 Configuration Guide

Custom Models & User Local Overrides

Track custom, fine-tuned, or local models by dropping a .yaml file into your project:

provider: "ollama"
models:
  deepseek-r1:
    input_cost_per_1m: 0.0
    output_cost_per_1m: 0.0

Pass the pricing directory:

client = CostOpt(OpenAI(), pricing_dir="./my_pricing")

🛡️ Security Audit

CostOpt has undergone automated penetration testing for SQL injections, CORS misconfigurations, and rate-limiting DB locks. See the full audit report at docs/SECURITY_AUDIT.md.

📄 License

This project is licensed under the MIT License. See LICENSE for details.