llm-token-guard 0.4.1

Track LLM token usage, enforce limits, and trigger alerts — across OpenAI, Groq, OpenRouter, AWS Bedrock, and custom providers.

TokenGuard

Production-ready token tracking, budget limits, and alerts for LLM applications.

---

⚡ Quick Start

# Sync exact tracking
from token_guard import TokenGuard
guard = TokenGuard(max_tokens=5_000)
result = guard.track_usage("alice", input_tokens=42, output_tokens=15)

print(result.total_tokens)                  # 57
print(result.limit_exceeded)                # False
print(result.cumulative_usage.total_tokens) # 57

# Async exact tracking
from token_guard import AsyncTokenGuard
async_guard = AsyncTokenGuard(max_tokens=5_000)
result = await async_guard.track_usage("bob", input_tokens=42, output_tokens=15)

print(result.total_tokens)                  # 57
print(result.limit_exceeded)                # False

---

Table of Contents

• Why TokenGuard?
• Architecture
• Features
• Installation
• Documentation Guides
  • Token Counting & Providers
  • Storage Backends
  • FastAPI Integration
  • Async Support
  • Custom Backends
• Provider Compatibility
• Project Structure
• Examples
• Running Tests
• Roadmap
• Contributing
• License

---

🧠 Why TokenGuard?

LLM calls are billed per token (inputs + outputs). Unchecked application consumption can quickly lead to unexpected cost spikes, upstream API rate-limiting blockages, or abuse.

TokenGuard acts as a lightweight, thread-safe, and event-loop-safe middleware layer. Use it to:

• Prevent Cost Spikes: Set and enforce strict token usage budgets per user/session.
• Unify Tracking: Track consumption across OpenAI, Groq, OpenRouter, and AWS Bedrock under a single API.
• Flexible Storage: Keep track in-memory (dev) or plug in Redis or SQLite (prod) with a single config change.
• Proactive Alerts: Fire warnings and webhook notifications (Slack, console) the instant thresholds are crossed.

---

🏗️ Architecture

TokenGuard splits concerns into clean interfaces:

• Counters: Tokenizer logic (tiktoken, HuggingFace transformers, or Bedrock count API).
• Storage: Persists cumulative records (Memory, Redis, SQLite).
• Alerts: Triggers limit-exceeded warning dispatches.

graph TD
    App[Application] --> TG[TokenGuard / AsyncTokenGuard]
    TG --> Counter[Token Counter]
    TG --> Storage[Storage Backend]
    TG --> Alerts[Alert Manager]
    TG --> Result[TrackResult]

---

✨ Features

Feature	Detail
Multi-Provider Counting	OpenAI (exact local), Groq, OpenRouter, AWS Bedrock
Exact Tracking	track_usage() records exact token metrics directly from API payloads
Pluggable Storage	Seamlessly swap backends (InMemory, Redis, SQLite) with one config line
Budget Enforcement	Track usage against configurable limits per user_id
Extensible Alerts	console, Slack, webhooks, or custom handlers
Auto-Detect Backend	Auto-detect model tokens based on model name strings
FastAPI & Async Ready	Full async entry points and async-native database integrations
Robust Test Suite	173 offline unit and integration tests

---

📦 Installation

# Core package (includes OpenAI/tiktoken local counting and memory storage)
pip install llm-token-guard

# Install optional backends & providers
pip install "llm-token-guard[redis]"         # Redis storage support
pip install "llm-token-guard[sqlite-async]"  # Async SQLite (aiosqlite) support
pip install "llm-token-guard[groq]"          # Groq HuggingFace tokenizers
pip install "llm-token-guard[bedrock]"       # AWS Bedrock boto3 exact counts
pip install "llm-token-guard[all]"           # All optional dependencies

---

📖 Documentation Guides

Advanced configuration, setup patterns, and code integrations are organized into individual guides:

1. Token Counting & Providers

• Tiktoken exact token counts for OpenAI and OpenRouter.
• HuggingFace tokenizer integrations for Groq models.
• AWS API-driven exact counting for AWS Bedrock.
• Provider accuracy comparison table.

2. Storage Backends

• Using default InMemoryStorage.
• Setting up connection pools, keys namespaces, and TTLs in RedisStorage.
• Configuring persistent file storage via SQLiteStorage.
• Initializing via Environment Variables or Configuration Dictionaries.

3. FastAPI Integration

• Adding AsyncTokenGuard to standard web applications.
• Managing exact API counts inside async routes without blocking.
• Guide to API commands (curl) for tracking, checking, and resetting.

4. Async Support

• Writing non-blocking async codebases with AsyncTokenGuard.
• Selecting async storage backends (AsyncInMemoryStorage, AsyncRedisStorage, AsyncSQLiteStorage).
• Configuring mixed sync/async alert triggers.

5. Custom Backends

• Subclassing BaseTokenCounter and registering with CounterFactory.
• Subclassing BaseStorage and registering with StorageFactory for databases (e.g. Postgres, DynamoDB).

---

📊 Provider Compatibility

Provider	Accuracy	Counting Method	Async Compatible	API Dependency
OpenAI	100% (Exact)	Local tiktoken	Yes	None
Groq (Default)	~95%	Local tiktoken cl100k	Yes	None
Groq (Transformers)	100% (Exact)	Local AutoTokenizer	Yes	transformers
AWS Bedrock (Local)	~85% - ~95%	Local estimator	Yes	None
AWS Bedrock (API)	100% (Exact)	AWS CountTokens API	Yes	boto3
OpenRouter	~85% - 100%	Local estimator	Yes	None
Direct Tracking	100% (Exact)	track_usage(input, output)	Yes	None

---

🏗️ Project Structure

token_guard/
├── docs/                 # Detailed guides and reference docs
├── token_guard/          # Core library source code
│   ├── counters/         # Token counters (OpenAI, Groq, etc.)
│   └── storage/          # Storage backends (Memory, Redis, SQLite)
├── tests/                # Test suite (sync & async)
├── example_fastapi.py    # FastAPI integration demo
└── pyproject.toml        # Build configuration and dependencies

---

🚀 Examples

Ready-to-run examples demonstrating different configuration patterns:

• FastAPI Integration: Async token limits and route handling.
• Multi-Provider Demo: Basic usage mapping different counter and storage backends.

---

🧪 Running Tests

pip install -e ".[dev]"

# Run all offline sync and async tests (no API keys required)
pytest tests/test_token_guard.py tests/test_storage.py tests/test_async_token_guard.py -v

# Run integration tests (requires GROQ_API_KEY env var)
export GROQ_API_KEY=gsk_...
pytest tests/test_groq_integration.py -v -s

# Run the complete test suite
pytest tests/ -v

---

🗺️ Roadmap

• Multi-provider token counting — OpenAI, Groq, OpenRouter, Bedrock ✅
• Auto-detect provider — CounterFactory.auto() ✅
• Pluggable storage — Memory, Redis, SQLite ✅
• StorageFactory — from_env(), from_url(), from_config() ✅
• Redis connection pooling + TTL + from_url() + ping() ✅
• GitHub Actions CI/CD — auto-publish on version tag ✅
• Exact token tracking — track_usage() with API-reported counts ✅
• Async support — async def track(...) for async frameworks ✅
• Sliding window limits — hourly / daily token budgets per user
• Budget warnings — alert at configurable % (e.g. 80%) before hard limit
• Per-model cost tracking — estimate USD cost alongside token counts
• Prometheus metrics — expose token_guard_tokens_total counter
• Vertex AI / Cohere — dedicated exact-count backends
• PostgreSQL / DynamoDB — built-in storage backends

---

🤝 Contributing

Contributions are welcome! Please follow these basic guidelines:

1. Fork the repository and create a feature branch.
2. Ensure the full test suite passes locally before submitting your PR:

   pytest tests/ -v
   

3. Follow PEP 8 style standards.

---

📄 License

MIT ©Abhijit Gunjal — see LICENSE for details.