tensorcost 0.4.0

One-line wrapper SDK for TensorCost — LLM observability and applied-mode inference routing for OpenAI and Anthropic clients.

tensorcost — Python SDK

One-line wrapper SDK for TensorCost. Adds fire-and-forget cost observability to any OpenAI or Anthropic Python client without changing how your code calls the underlying provider.

Install

pip install tensorcost

Runtime dependencies: httpx only.

One-line example

from openai import OpenAI
from tensorcost import wrap

client = wrap(OpenAI(api_key="sk-..."))

# use `client` exactly like a normal OpenAI client.
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "hello"}],
)

Anthropic works identically:

from anthropic import Anthropic
from tensorcost import wrap

client = wrap(Anthropic(api_key="sk-ant-..."))
client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=128,
    messages=[{"role": "user", "content": "hello"}],
)

Configuration

wrap() resolves config in this order:

1. Explicit kwargs to wrap(client, api_key=..., base_url=..., tenant_id=...)
2. Environment variables: TENSORCOST_API_KEY, TENSORCOST_BASE_URL, TENSORCOST_TENANT_ID, TENSORCOST_PROXY_URL
3. Defaults: base_url defaults to https://api.tensorcost.com

If no api_key is found, wrap() raises MissingConfigError. Same if applied_mode=True and no proxy_url is available from either source. Both are the only failure modes the SDK does NOT swallow.

Applied mode (Layer 2)

By default the SDK is observe-only: it fires telemetry after your call completes, and your call goes directly to OpenAI / Anthropic. Applied mode routes chat.completions (and Anthropic messages.create) through the TensorCost inference-proxy, which can redirect calls to a different provider based on per-tenant policy.

from openai import OpenAI
from tensorcost import wrap

client = wrap(
    OpenAI(api_key="sk-..."),
    api_key="tc-...",              # TensorCost API key
    applied_mode=True,
    proxy_url="https://api.my-instance.tensorcost.com",
    # or set TENSORCOST_PROXY_URL in the environment
)

# Use client exactly as before — the proxy is transparent.
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "hello"}],
)

The SDK forwards each provider's native request body unchanged. OpenAI calls go to the proxy as OpenAI-shaped JSON; Anthropic calls go as Anthropic-shaped JSON. The proxy identifies the source from the x-tc-provider-url base URL and handles any translation internally. This means the proxy response also comes back in the upstream provider's native shape — your existing parsing code sees no difference.

The proxy decides per-request whether to route or pass through. If the proxy call fails for any reason the SDK falls back to the direct provider call and logs a warning — the customer's call always completes.

Telemetry (observe-mode) still fires on every call regardless of applied_mode. The two paths are independent.

For the proxy design see docs/features/inference-proxy-applied-mode.md.

Applied-mode hardening (v0.4.0)

When applied_mode=True, you can configure retry policy, timeouts, lifecycle hooks, and the fail-open circuit breaker.

Retries

from tensorcost import wrap, RetryConfig

client = wrap(
    OpenAI(api_key="sk-..."),
    applied_mode=True,
    proxy_url="https://...",
    retry=RetryConfig(max_attempts=5, base_delay_ms=500.0, max_delay_ms=30_000.0),
)

5xx responses, network errors, and 429 are retried with jittered exponential backoff. A Retry-After header on 429 is honoured. 4xx errors (except 429) are never retried.

Timeouts

client = wrap(
    OpenAI(api_key="sk-..."),
    applied_mode=True,
    proxy_url="https://...",
    timeout_s=30.0,   # default 60.0
)

Surfaces as TensorCostTimeoutError when exceeded.

Lifecycle telemetry hooks

from tensorcost import wrap, LifecycleEvent

def on_event(event: LifecycleEvent) -> None:
    # event.kind is one of:
    #   "before_request" | "after_response" | "on_retry"
    #   | "on_error" | "on_fallback"
    print(event.kind, event.model, event.attempt_number)

client = wrap(
    OpenAI(api_key="sk-..."),
    applied_mode=True,
    proxy_url="https://...",
    on_lifecycle_event=on_event,
)

Events contain only request metadata — never prompt content, response bodies, or credentials. Errors raised by the callback are swallowed.

Typed errors

Every failure from the proxy path is a subclass of TensorCostError:

from tensorcost import (
    TensorCostError,
    TensorCostNetworkError,
    TensorCostTimeoutError,
    TensorCostProxyError,
    TensorCostQuotaError,
    TensorCostProviderError,
)

try:
    resp = client.chat.completions.create(model="gpt-4o-mini", messages=[])
except TensorCostQuotaError as e:
    print("rate limited; retry after", e.retry_after_ms, "ms")
except TensorCostError as e:
    print("tensorcost error", e.status, e.attempt)

Circuit breaker

After 3 consecutive proxy 5xx responses the circuit opens and requests route directly to the provider (fail-open). The circuit closes after 5 consecutive probe successes.

Set fail_open_enabled=False if direct provider access is unavailable (e.g. in-VPC deployments) — proxy failures then surface as TensorCostProxyError instead of falling back.

client = wrap(
    OpenAI(api_key="sk-..."),
    applied_mode=True,
    proxy_url="https://...",
    fail_open_enabled=False,   # raise on proxy error instead of falling through
)

Fail-open guarantee

If TensorCost is unreachable, slow, or returns an error, your underlying provider call still completes normally. The SDK logs a warning via logging.getLogger("tensorcost") and moves on. We will never break your production traffic.

What gets sent

For every intercepted call we POST a JSON document with:

• SDK version, provider name, model, operation
• Request and response timestamps
• prompt_tokens / completion_tokens (or Anthropic equivalents) from the provider's usage field
• A correlation UUID
• The status (success / error) and an error message if the call failed

We do NOT capture prompt content, completion content, or any user data. Cost calculation happens server-side.

Authentication uses a short-lived JWT obtained via POST /api/inference-proxy/sdk-token/exchange. The long-lived API key never leaves the SDK process after the first exchange and is cached in memory.

Currently supported

• OpenAI (openai >= 1.0) — chat.completions.create, completions.create
• Anthropic (anthropic >= 0.20) — messages.create

Coming soon

• AWS Bedrock (on the roadmap)
• Azure OpenAI (on the roadmap)
• Google Vertex AI (v1.0)
• Streaming responses (v1.0)
• Tool / function calling (v1.0)
• Async clients (AsyncOpenAI, AsyncAnthropic) (v1.0)
• LangChain / LlamaIndex adapters (v1.0)

Development

pip install -e '.[dev]'
pytest

License

Apache-2.0. See LICENSE for the full text.