tensorcost — Python SDK

One-line wrapper SDK for TensorCost. Adds fire-and-forget cost observability and optional inference control to OpenAI, Anthropic, AWS Bedrock, and Google Vertex AI clients without changing how your code calls the underlying provider.

Install

pip install tensorcost

Runtime dependencies: httpx only.

One-line example

from openai import OpenAI
from tensorcost import wrap

client = wrap(OpenAI(api_key="sk-..."))

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "hello"}],
)

Anthropic works identically:

from anthropic import Anthropic
from tensorcost import wrap

client = wrap(Anthropic(api_key="sk-ant-..."))
client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=128,
    messages=[{"role": "user", "content": "hello"}],
)

Configuration

wrap() resolves config in this order:

1. Explicit kwargs to wrap(client, api_key=..., base_url=..., ...)
2. Environment variables: TENSORCOST_API_KEY, TENSORCOST_BASE_URL, TENSORCOST_TENANT_ID, TENSORCOST_PROXY_URL, etc.
3. Defaults: base_url defaults to https://api.tensorcost.com

If no api_key is found, wrap() raises MissingConfigError. Same if max_layer is steer or route and no proxy_url is available.

Control layers (max_layer)

v1.0 replaces the old applied_mode boolean with a control-layer ceiling. A layer is a ceiling, not a mode — the effective layer is always min(your max_layer, console published layer).

from openai import OpenAI
from tensorcost import wrap

client = wrap(
    OpenAI(api_key="sk-..."),
    api_key="tc-...",
    max_layer="route",
    proxy_url="https://api.my-instance.tensorcost.com",
)

The SDK fetches the console-published ceiling from GET /api/inference-proxy/v1/sdk-layer (30s cache, decays to govern after 5 minutes unreachable). Govern refusals and proxy decisions surface in the x-tc-decision response header.

applied_mode=True still works but is deprecated — it maps to max_layer="route" with a warning. Prefer max_layer in new code.

Proxy hardening

When max_layer is govern, steer, or route, you can configure retries, timeouts, lifecycle hooks, and the fail-open circuit breaker.

from tensorcost import wrap, RetryConfig

client = wrap(
    OpenAI(api_key="sk-..."),
    max_layer="route",
    proxy_url="https://...",
    retry=RetryConfig(max_attempts=5, base_delay_ms=500.0, max_delay_ms=30_000.0),
    timeout_s=30.0,
    fail_open_enabled=True,
)

Typed errors: TensorCostError, TensorCostNetworkError, TensorCostTimeoutError, TensorCostProxyError, TensorCostQuotaError, TensorCostProviderError.

Agent-run attribution

client = wrap(
    OpenAI(api_key="sk-..."),
    agent_id="support-bot",
)

task_client = client.with_meta(workflow_id="ticket-48291")
task_client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this ticket"}],
)

Applied-mode proxy calls forward x-tc-agent-id / x-tc-workflow-id. with_meta() is supported on OpenAI and Anthropic wrapped clients; Bedrock and Vertex accept wrap-time defaults only.

Fail-open guarantee

If TensorCost is unreachable, slow, or returns an error, your underlying provider call still completes normally. The SDK logs a warning via logging.getLogger("tensorcost") and moves on.

What gets sent

For every intercepted call we POST metadata only — never prompt or completion content:

• SDK version, provider, model, operation, modality
• Timestamps, token counts, correlation UUID, status
• Environment, agent/workflow ids, chargeback tags when configured

Authentication uses a short-lived JWT from POST /api/inference-proxy/sdk-token/exchange.

Currently supported

• OpenAI (openai >= 1.0) — chat, completions, embeddings, responses
• Anthropic (anthropic >= 0.20) — messages
• AWS Bedrock (boto3 bedrock-runtime) — invoke/converse (+ streaming placeholders). Observe/govern only — max_layer="steer" or "route" raises MissingConfigError.
• Google Vertex AI (google-genai with vertexai=True) — generate_content (+ streaming). Observe/govern only.
• Azure OpenAI — wrap AzureOpenAI (tagged provider: openai)

Development

pip install -e '.[dev]'
pytest

See PUBLISHING.md for the PyPI release runbook.

License

Apache-2.0. See LICENSE for the full text.