tokensense-ai 0.2.3

LLM cost observability and smart routing — stays in your code, works everywhere.

Visual Vortex

TokenSense

by Visual Vortex

LLM cost observability and smart routing — stays in your code, works everywhere.

---

pip install tokensense-ai

from tokensense import observe
import anthropic

client = observe(anthropic.Anthropic())
response = client.messages.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Hello"}]
)
# → model=claude-sonnet-4-6 | in=12 out=24 tokens | $0.0003 | 847ms

That's it. One import. One wrapper. Every LLM call you make is now tracked.

---

Why TokenSense?

Other observability tools make you choose between easy and private. A proxy is easy but your API keys and prompts go through someone else's server. A self-hosted platform is private but takes 30 minutes to set up before you see anything.

TokenSense does neither. It runs inside your process. Nothing leaves unless you tell it to.

• Privacy by default — prompts never leave your process. Ever. Not to us, not to anyone.
• Zero latency — the observer fires after the response returns. Your call time is unchanged.
• Works everywhere — dev, prod, Docker, Lambda, K8s. No infra required.
• Smart routing — context-budget-aware model switching with automatic escalation on failure.
• Framework agnostic — wraps OpenAI, Anthropic, Groq directly. Not tied to LangChain or anything else.
• You own the data — output to your terminal, SQLite, your existing logger, or any HTTP endpoint.

---

Quickstart

Observe any LLM client

from tokensense import observe
import openai

# OpenAI
client = observe(openai.OpenAI())

# Anthropic
client = observe(anthropic.Anthropic())

# Groq
client = observe(groq.Groq())

# your existing code stays exactly the same
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain async/await"}]
)
# → model=gpt-4o-mini | in=18 out=312 tokens | $0.0001 | 623ms

Change where data goes

from tokensense import observe
from tokensense.outputs import SQLite, Logger, HTTP

# save locally — great for dev
client = observe(anthropic.Anthropic(), output=SQLite("./usage.db"))

# write to Python logger — goes to CloudWatch, Datadog, whatever you already use
client = observe(anthropic.Anthropic(), output=Logger("tokensense"))

# post to your own endpoint
client = observe(anthropic.Anthropic(), output=HTTP("https://your-server.com/ingest"))

Default output when you don't specify anything:

• In development → prints to terminal + writes to ./tokensense.db
• In production (ENV=production) → writes to Python logger, goes to your existing log infra

Add metadata per call

client = observe(
    anthropic.Anthropic(),
    user_id="user_123",
    session_id="sess_abc",
    tags=["production", "chat"],
)

Smart routing

from tokensense import Router, Rule

router = Router(
    tiers={
        "small":  ["llama-3.1-8b-instant", "claude-haiku-4-5"],
        "large":  ["claude-sonnet-4-6", "gpt-4o"],
    },
    rules=[
        # never route to small model if history is long
        Rule(if_context_tokens_gt=4000, deny_tiers=["small"]),

        # pin sensitive tasks to large model always
        Rule(if_task="legal-review", pin_tier="large"),
    ]
)

decision = router.route(
    messages=[{"role": "user", "content": "Review this contract"}],
    task_hint="legal-review"
)

print(decision.model) # → 'claude-sonnet-4-6'
print(decision.reason) # → 'pinned to large by rule'

---

ShadowTest

Stop guessing if a cheaper model will ruin your app. ShadowTest runs your real prompts against multiple models in parallel, scoring the results before you deploy.

from tokensense import ShadowTest, observe

test = ShadowTest(
    clients={
        "current": observe(anthropic.Anthropic()),
        "candidate": observe(openai.OpenAI())
    },
    prompts=[
        {
            "messages": [{"role": "user", "content": "Return the user profile as JSON."}],
            "model_current": "claude-3-5-sonnet",
            "model_candidate": "gpt-4o-mini",
            "expected_format": "json"
        }
    ],
    scoring="format-check" # Also supports: exact-match, similarity, llm-judge
)

report = test.run()
print(report.summary())
# → Tier        Pass Rate    Avg Cost     Avg Latency 
# → current     100%         $0.0030      800ms
# → candidate   95%          $0.0001      300ms

---

What gets logged

By default, TokenSense logs only metadata — never your prompt content or response text.

Field	Logged by default	Opt-in
Model name	✅
Input tokens	✅
Output tokens	✅
Cost (USD)	✅
Latency (ms)	✅
Routing decision	✅
Status (success/error)	✅
Prompt content	❌	log_prompts=True
Response content	❌	log_responses=True
User ID / tags	❌	pass as kwargs

To enable prompt logging explicitly:

client = observe(anthropic.Anthropic(), log_prompts=True, log_responses=True)

---

Output options

Output	Best for	Setup
Stdout()	Development, debugging	None
SQLite(path)	Local persistence, single process	None
Logger(name)	Production — any log infra	None
HTTP(url)	Custom server, self-hosted dashboard	Your endpoint

from tokensense.outputs import Stdout, SQLite, Logger, HTTP

# chain multiple outputs
from tokensense.outputs import Multi

client = observe(
    anthropic.Anthropic(),
    output=Multi(Stdout(), SQLite("./usage.db"))
)

---

Supported providers

Provider	Sync	Async	Streaming
Anthropic	✅	✅	✅
OpenAI	✅	✅	✅
Groq	✅	✅	✅
Google Gemini	✅	✅	✅
LiteLLM	✅	✅	✅

---

Integrations

TokenSense has native integrations with LangChain and LlamaIndex via callback handlers. You do not need to use observe(), just pass the callback handler into your LLM configuration.

LangChain

from tokensense import TokenSenseCallbackHandler
from langchain_groq import ChatGroq

llm = ChatGroq(model="llama-3.1-8b-instant")
response = llm.invoke("Hello!", config={"callbacks": [TokenSenseCallbackHandler()]})

LlamaIndex

from tokensense import TokenSenseLlamaIndexCallback
from llama_index.core.callbacks import CallbackManager
from llama_index.core import Settings
from llama_index.llms.groq import Groq

Settings.callback_manager = CallbackManager([TokenSenseLlamaIndexCallback()])
llm = Groq(model="llama-3.1-8b-instant")
response = llm.complete("Hello!")

---

Semantic Caching

Stop paying for identical prompts. TokenSense includes a local, sqlite-vec powered semantic cache that intercepts duplicate requests before they hit the LLM.

from tokensense import observe
from tokensense.cache import SQLiteVectorCache

cache = SQLiteVectorCache("./tokensense.db")
client = observe(openai.OpenAI(), cache=cache)

# First call: hits OpenAI API and costs money
client.chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": "Explain async/await"}])

# Second call: intercepted instantly, latency drops to ~1ms, cost $0.00
client.chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": "Explain async/await"}])

---

CLI Tools

TokenSense comes with built-in CLI commands to manage your database and pricing.

View Total Spend:

tokensense report

Aggregates your SQLite database and prints total token usage and USD spent per model.

Update Model Pricing:

tokensense update-prices

Downloads the latest live pricing database from the open-source LiteLLM project so you're always accurate, even for models released today.

---

OpenTelemetry (OTEL) Export

Enterprise ready. Export your traces natively to Datadog, Grafana, or Jaeger.

from tokensense import observe
from tokensense.outputs import Multi, OTEL

client = observe(
    anthropic.Anthropic(),
    output=Multi(SQLite("./usage.db"), OTEL(service_name="my-app"))
)

---

No telemetry. Ever.

TokenSense does not call home. There is no usage tracking, no anonymous analytics, no background pings to any server. The framework has no network dependency of its own.

Read PRIVACY.md for exactly what is and isn't captured.

---

Roadmap

• observe() wrapper — OpenAI, Anthropic, Groq
• Stdout, SQLite, Logger, HTTP outputs
• Auto environment detection (dev vs prod)
• Router with context budget check
• Shadow testing before switching tiers
• tokensense report CLI — spend summary in terminal
• OpenTelemetry (OTEL) Export — native integration with Datadog/Grafana
• Local Semantic Caching — via SQLite vector extension to save cost/latency
• LiteLLM Integration — native support for observing litellm.completion()
• Google Gemini support
• LangChain integration
• LlamaIndex integration

---

Contributing

TokenSense is MIT licensed and built in public. Issues, PRs, and feedback are welcome.

git clone https://github.com/SahilSelokar/Tokensense
cd Tokensense
pip install -e ".[dev]"
pytest

See CONTRIBUTING.md for guidelines.

---

License

MIT — see LICENSE

---

Built by Visual Vortex