token-overflow 0.1.0

Prevent LLM context window overflow with token counting, smart truncation, and budget management

token-overflow

Prevent LLM context window overflow. Count tokens, enforce budgets, and smart-truncate prompts before they silently fail.

The Problem

LLM APIs silently truncate or error when you exceed context windows. You don't know until you get garbage output or a 400 error. token-overflow catches this before the API call.

Install

pip install token-overflow

Quick Start

from token_overflow import TokenGuard, fit

# Simple: fit text into a budget
short = fit("very long document...", max_tokens=1000)

# Guard a full prompt
guard = TokenGuard(model="gpt-4")  # auto-detects 8192 limit

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": huge_document},
]

# Check before calling API
if guard.would_overflow(messages):
    messages = guard.truncate(messages, strategy="tail")

# Or one-liner: safe messages guaranteed to fit
safe = guard.safe(messages)

Token Counting

from token_overflow import count_tokens, count_messages

# Count tokens in text
n = count_tokens("Hello world", model="gpt-4")  # 2

# Count tokens in chat messages
n = count_messages([
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Hi"},
], model="gpt-4")

Truncation Strategies

guard = TokenGuard(model="gpt-4")

# Keep the tail (most recent), trim from head
guard.truncate(messages, strategy="tail")

# Keep the head (system prompt), trim from tail
guard.truncate(messages, strategy="head")

# Keep head and tail, trim middle
guard.truncate(messages, strategy="middle")

# Custom reserve: keep 500 tokens for response
guard.truncate(messages, strategy="tail", reserve=500)

Model Limits

from token_overflow import get_limit

get_limit("gpt-4")           # 8192
get_limit("gpt-4-turbo")     # 128000
get_limit("gpt-3.5-turbo")   # 16385
get_limit("claude-3-opus")   # 200000
get_limit("claude-3-sonnet") # 200000

Features

• Zero config: Knows context limits for 30+ models
• Multiple strategies: head, tail, middle truncation
• Message-aware: Understands chat message format, preserves system prompts
• Reserve tokens: Leave room for the response
• Fast counting: Uses tiktoken when available, falls back to approximation
• No dependencies required: tiktoken optional for exact counts

License

MIT