costhelm 0.1.0

Self-hosted cost-governance gateway for LLM APIs: per-model pricing, principal attribution, hard budget enforcement, cost/quality routing, semantic caching, and OTel GenAI observability.

costhelm

A self-hosted cost-governance gateway for LLM APIs. Your agents talk to one local HTTP endpoint; costhelm owns the provider keys and answers every call with the numbers that actually matter: what it cost, who spent it, and how much budget is left.

Most gateways treat cost as a report you read after the money is gone. costhelm treats it as a precondition: every call is projected at worst-case cost and admitted against a hard budget before any provider is contacted. A refusal is an HTTP 402 with the arithmetic in the body — limit, spend, projection, shortfall — and it costs $0, because nothing was sent.

call 4: ollama/phi4:latest — $0.017000 (total $0.047000) — 'Running budget checks before...'
call 5: REFUSED — session:budget-agent-demo would exceed its lifetime budget:
        spent $0.047000 of $0.050000, this call projects $0.016000

Features

Hard budgets	Per-principal ceilings (tenant/project/user/agent/session, glob-able) with minute/hour/day/month/lifetime periods. Pre-call admission on a worst-case projection; spend is always re-derived from the ledger, never a parallel counter. Runtime overrides via POST /v1/budget — clamp a runaway session without a deploy.
Priced ledger	One SQLite row per call, priced from a per-model pricing.yaml (glob patterns, cache-read/write and batch multipliers) and attributed to all five principal dimensions. CI fails if a configured default model would price at $0.
Tier routing + cascade	A small router LLM classifies each prompt (TINY/LARGE/HUGE); the caller's declared role clamps the tier with floors and ceilings; candidates are ordered by cost, quality, or a tradeoff dial. Low-confidence answers escalate one tier — confidence is scored from structural signals, never a judge LLM.
Provider failover	Eight providers behind one API (OpenAI, Gemini with a numbered key pool, OpenRouter, Ollama, Groq, Cerebras, NVIDIA, GitHub Models). RPM/RPD/TPM pacing, capability-aware picking (tools/vision/reasoning/structured), failure-class cooldowns — all configurable in routing.yaml.
Semantic cache	Embed → cosine match → skip the provider entirely. A hit bills $0 and records the tokens as saved; truncated and empty completions are refused into the cache. Off by default, one line to enable.
Observability	OTel spans with gen_ai.* attributes plus cost; two zero-build HTML dashboards; /v1/refusals captures the 402s that by design leave no span and no ledger row. Prompt/completion capture is off by default.
Optional auth	One env var (COSTHELM_API_KEYS) guards /v1/* with bearer or X-API-Key. Off by default for localhost.

Quickstart

uvx costhelm serve

(or from a clone: uv sync && uv run costhelm serve)

Add a provider key to .env (see .env.example) — or none at all if Ollama is running locally:

curl -s localhost:8111/v1/chat -H 'Content-Type: application/json' \
  -d '{"prompt": "hello", "max_tokens": 50}'

Every response carries cost, budget, cache, and router_decision envelopes. Open http://localhost:8111/ for the dashboard.

To see budget enforcement end-to-end, run the example agent — it arms a $0.05 ceiling, spends it, and prints the 402:

uv run python examples/budget_agent.py

How a request flows

POST /v1/chat
  │
  1. build_context      normalise, estimate tokens, resolve principal, agent pin
  2. semantic cache     hit → return stored answer, bill $0, record tokens saved
  3. select_candidates  router-LLM tier classify → role clamp → cost/quality order
  4. dispatch           ┌ cascade loop ──────────────────────────────┐
                        │  for each candidate:                       │
                        │    budget admission (worst-case, pre-call) │──402
                        │    provider call (+1 retry, backoff)       │
                        │    structured-output validate/repair       │
                        │    confidence check → maybe escalate tier ─┘
                        │    meter → priced ledger row, OTel span
                        └ return text + cost/budget/cache envelopes

Each stage is a module in costhelm/pipeline/, and each is inert unless configured: with the shipped config files, costhelm behaves like a plain failover gateway.

Configuration

Four YAML files, three-layer resolution — an env-var path override wins, then ~/.costhelm/<name>.yaml, then the packaged default. costhelm paths prints what resolved where.

File	Governs
pricing.yaml	$/Mtok per model, cache/batch multipliers, quality weights
budgets.yaml	Spend ceilings per principal (ships empty: nothing is refused)
routing.yaml	Tiers, roles, ladders, escalation, provider limits, agent pins
cache.yaml	Semantic cache (ships disabled)

Secrets and switches live in .env — see .env.example. A provider registers only if its key variable is set.

Client

from costhelm.client import CosthelmClient, BudgetExceeded

async with CosthelmClient() as gw:
    await gw.set_budget("session:run-42", 0.05, period="day")
    try:
        r = await gw.chat("hello", request={"session": "run-42", "auto_route": "worker"})
        print(r["text"], r["cost"]["total_usd"])
    except BudgetExceeded as e:
        print("refused:", e.envelope["shortfall_usd"])

The wire contract is plain HTTP JSON — any language works. The client holds no provider credential; that boundary is the point.

Deployment

deploy/ has systemd/launchd/Windows-service templates, a Modal wrapper, and a docker-compose file for a local Jaeger to receive the OTel spans. Docs: https://costhelm.com

Origins

costhelm grew out of an agentic-AI course project (a gateway serving a budget-aware agent runtime) and was extracted and restructured into a standalone framework.

License

MIT