llmguard — A Lightweight Python Decorator for Reliable LLM API Calls

A lightweight Python library that wraps any LLM API call (or any external, flaky, rate-limited API) with response caching, token-bucket rate limiting, automatic retries with exponential backoff, and usage statistics using a single decorator.

Instead of repeatedly solving these problems in every project, llmguard provides a reusable, production-friendly solution with a clean and simple API.

---

Why I Built This

While developing my AI Resume Matcher project using the Gemini API, I repeatedly encountered the same issues during development:

• Running the same prompt multiple times consumed API credits unnecessarily.
• Temporary API failures interrupted requests.
• Rate limits caused delays and failed requests.
• There was no simple way to measure how much caching was reducing API usage.

Rather than solving these problems inside one application, I extracted them into a reusable Python package that can be integrated into any LLM-powered project.

---

Features

• 🚀 Simple @llmguard decorator
• ⚡ In-memory TTL response caching
• 🔁 Automatic retries with exponential backoff and jitter
• 🪣 Token-bucket rate limiting
• 📊 Built-in usage statistics
• 💰 Estimated API cost savings
• 🖥️ Command-line interface for viewing statistics
• 🧪 Comprehensive unit tests
• 📦 Packaged as a reusable Python library

---

Installation

Install from source

git clone https://github.com/aadhya-code/llmguard.git
cd llmguard
pip install -e .

Build the package

python -m build

---

Quick Start

from llmguard import llmguard

@llmguard(
    cache_ttl=3600,
    max_calls_per_period=10,
    period_seconds=60,
    max_retries=3,
    cost_per_call=0.002
)
def ask_llm(prompt: str) -> str:
    return call_some_llm_api(prompt)

# First call → API request
response = ask_llm("Explain Retrieval-Augmented Generation")

# Second identical call → Served from cache
response = ask_llm("Explain Retrieval-Augmented Generation")

---

Example

from llmguard import llmguard

@llmguard(
    cache_ttl=1800,
    max_calls_per_period=30,
    period_seconds=60,
    max_retries=5,
)
def generate(prompt):
    return llm.generate(prompt)

print(generate("Summarise this research paper"))

---

CLI Usage

View usage statistics:

llmguard-stats

Example output:

llmguard usage stats
================================
Total wrapped calls:     47
Cache hits:              19
Cache misses:            28
Cache hit rate:          40.4%
Total retries used:      3
Rate-limit waits:        5
Estimated $ saved:       $0.0380

Reset statistics:

llmguard-stats --reset

---

How It Works

Response Caching

• Generates a SHA-256 hash using the function name and arguments.
• Returns cached responses for identical requests.
• Supports configurable Time-To-Live (TTL) expiration.

Token-Bucket Rate Limiting

• Smoothly limits request rate.
• Prevents bursts that commonly occur with fixed-window rate limiters.
• Blocks only when necessary.

Automatic Retry

• Retries failed requests using exponential backoff.
• Adds random jitter to prevent retry storms.
• Raises a descriptive exception if all retry attempts fail.

Usage Statistics

Records:

• Total wrapped calls
• Cache hits
• Cache misses
• Retry count
• Rate-limit waits
• Estimated API cost savings

Statistics are persisted locally so they remain available across different program executions.

---

Project Structure

llmguard/
│
├── llmguard/
│   ├── __init__.py
│   ├── cache.py
│   ├── retry.py
│   ├── rate_limiter.py
│   ├── guard.py
│   ├── stats.py
│   └── cli.py
│
├── tests/
│   ├── test_cache.py
│   ├── test_retry.py
│   ├── test_rate_limiter.py
│   └── test_guard.py
│
├── pyproject.toml
├── README.md
└── LICENSE

---

Architecture

                    User Function
                          │
                          ▼
                   @llmguard(...)
                          │
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
      TTL Cache     Retry Handler    Token Bucket
          │               │               │
          └───────────────┼───────────────┘
                          ▼
                     External LLM API

---

Running Tests

Run all tests:

pytest -v

Generate a coverage report:

pytest --cov=llmguard

---

Version

Current Version:

v0.1.0

---

Future Improvements

• Async (async/await) support
• Persistent cache backend (SQLite/Redis)
• Additional provider examples (Gemini, OpenAI, Anthropic)
• GitHub Actions CI/CD
• Benchmark suite
• Configurable logging

---

License

This project is licensed under the MIT License.

---

Contributing

Contributions, suggestions, and bug reports are welcome.

If you find an issue or have an improvement, feel free to open an issue or submit a pull request.