infertrack 1.0.3

Zero-config LLM call interceptor: cost, latency, and budget enforcement

infertrack

Zero-config LLM call interceptor. Track token usage, cost, and latency — locally, no cloud required.

pip install infertrack

from infertrack import watchdog

@watchdog(tag="summarise")
def ask(prompt):
    return client.chat.completions.create(
        model="qwen2.5:0.5b",
        messages=[{"role": "user", "content": prompt}]
    )

ask("Summarise this document...")

$ watchdog summary
  infertrack summary  ·  Last 24h
────────────────────────────────────────────────────────────────────────
  Calls                  47
  Input tokens         82.3k
  Output tokens        46.1k
  Total tokens        128.4k
  Total cost          $0.4100
  Avg latency           834ms
  Success rate         97.9%
────────────────────────────────────────────────────────────────────────
  Models:
    gpt-4o                                   31 calls
    qwen2.5:0.5b                             12 calls
    gpt-4o-mini                               4 calls

---

Why infertrack?

You're building an LLM feature. You want to know: how many tokens did each call use, how long did it take, what did it cost, did it fail silently? Right now you add print statements. Or you install LangSmith and spend a day configuring it.

infertrack gives you a decorator and a CLI. No cloud account. No config files. No framework lock-in.

	infertrack	LangSmith	Helicone
pip install + done	✓	✗	✗
No cloud account	✓	✗	✗
No traffic proxying	✓	✓	✗
Works with Ollama	✓	partial	✗
Budget enforcement	✓	✗	✗

---

Installation

pip install infertrack          # core + CLI

Requires: Python 3.10+, click>=8.0, openai>=1.0 (optional, for OpenAI/Ollama)

Works with Ollama out of the box — free, local, no API key:

ollama pull qwen2.5:0.5b

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

---

Usage

@watchdog decorator

Wrap any function that returns a raw API response object:

from infertrack import watchdog
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

@watchdog(tag="summarise", user_id="alice")
def ask(prompt: str):
    return client.chat.completions.create(
        model="qwen2.5:0.5b",
        messages=[{"role": "user", "content": prompt}]
    )

response = ask("What is the capital of France?")
print(response.choices[0].message.content)
# Every call is now logged to ~/.infertrack/logs.db

Parameters:

Parameter	Type	Description
tag	str | None	Label for grouping calls in the CLI
user_id	str | None	Per-user identifier, used by Budget enforcement
session_id	str | None	Optional session grouping label
db_path	Path | None	Override default DB path (useful in tests)

---

watch() context manager

Track one or more API calls in a block and read live metrics afterward:

from infertrack import watch

with watch(tag="batch-summarise", user_id="alice") as w:
    for doc in documents:
        resp = client.chat.completions.create(
            model="qwen2.5:0.5b",
            messages=[{"role": "user", "content": doc}]
        )
        w.add_response(resp)

print(f"Used {w.tokens_used:,} tokens")
print(f"Cost: ${w.cost_usd:.6f}")
print(f"Latency: {w.latency_ms:.0f}ms")
print(f"Calls: {w.call_count}")

---

Budget() enforcement

Stop a runaway loop before it empties your wallet:

from infertrack import Budget, BudgetExceeded

try:
    with Budget(max_usd=0.10, user_id="alice") as b:
        for chunk in large_document_chunks:
            resp = client.chat.completions.create(
                model="gpt-4o",
                messages=[{"role": "user", "content": chunk}]
            )
            b.add_response(resp)   # raises BudgetExceeded if over $0.10
except BudgetExceeded as e:
    print(f"Stopped: {e.user_id} spent ${e.spent:.4f} (limit ${e.limit:.4f})")

Period options — control what prior spend counts against the budget:

Budget(max_usd=0.10, user_id="alice", period="today")    # default: since midnight UTC
Budget(max_usd=1.00, user_id="alice", period="all")      # all historical spend
Budget(max_usd=0.05, user_id="alice", period="session")  # this block only

FastAPI example:

@app.post("/generate")
async def generate(prompt: str, user_id: str):
    with Budget(max_usd=0.05, user_id=user_id) as b:
        resp = client.chat.completions.create(...)
        b.add_response(resp)
    return {"result": resp.choices[0].message.content}

---

CLI

All commands read from ~/.infertrack/logs.db by default. Override with --db /path/to/logs.db.

watchdog summary

watchdog summary                   # last 24h
watchdog summary --last 7d         # last 7 days
watchdog summary --last all        # all time

watchdog tail

watchdog tail                      # last 20 calls
watchdog tail -n 50                # last 50 calls
watchdog tail --tag summarise      # filter by tag
watchdog tail --user alice         # filter by user
watchdog tail --model gpt-4o       # filter by model

watchdog top

watchdog top                              # top tags by cost
watchdog top --by tokens                  # rank by token usage
watchdog top --by calls                   # rank by call count
watchdog top --group user                 # group by user_id
watchdog top --group model                # group by model
watchdog top --limit 5 --last 7d          # top 5 in last 7 days

---

Storage

All data is stored locally in ~/.infertrack/logs.db (SQLite). No data ever leaves your machine.

Override the path with the --db flag on any CLI command, or pass db_path= to any decorator/context manager.

Each log entry contains:

Field	Description
id	UUID
timestamp	UTC datetime
provider	openai / anthropic / unknown
model	Model name from API response
input_tokens	From response.usage.prompt_tokens
output_tokens	From response.usage.completion_tokens
cost_usd	Calculated from embedded pricing table
latency_ms	Wall-clock milliseconds
success	True / False
tag	Optional label
user_id	Optional user identifier
session_id	Optional session identifier
error_msg	Exception message on failure

---

Supported models & pricing

Pricing is embedded in pricing/prices.json and used for cost calculation. Ollama models are always free.

Provider	Models
Ollama	qwen2.5, llama3.x, phi3.5/4, mistral, gemma2, deepseek-r1, … (free)
OpenAI	gpt-4o, gpt-4o-mini, gpt-4-turbo, o1, o3-mini, …
Anthropic	claude-3-5-sonnet, claude-3-5-haiku, claude-3-opus, …

Unknown models default to $0.00 cost and are still tracked for tokens and latency.

---

What infertrack does NOT do

• No prompt storage — the content of your messages is never logged
• No cloud sync — all data stays in your local SQLite file
• No network calls — zero outbound connections, ever
• No token counting library — tokens come from response.usage only (what the API reports)
• No framework required — works standalone alongside LangChain, LlamaIndex, or raw API calls

---

Roadmap

• @watchdog decorator
• watch() context manager
• Budget() enforcement
• CLI: summary, tail, top
• infertrack.intercept() — zero-code-change global monkey-patch (Day 7)
• Anthropic provider (Day 8)
• watchdog export --format csv/json (Day 9)
• Streaming response support (Day 10)

---

Development

git clone https://github.com/yourname/infertrack
cd infertrack
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Run tests (no Ollama needed)
pytest tests/unit/ -v

# Run integration tests (requires Ollama running)
pytest tests/integration/ -v -m integration

# Lint
ruff check src/

---

License

Apache