Skip to main content

django-llm-cache ⚡

PyPI version Python Django License: MIT Code style: black

A production-grade, modular, drop-in caching layer for LLM prompts and completions in Django applications.

Stop burning thousands of dollars every month on duplicate LLM API tokens across OpenAI, Gemini, Groq, and Anthropic.


💡 The Problem: The Duplicate Token Burn

Modern AI-enabled web apps, SaaS platforms, and enterprise dashboards suffer from a hidden margin killer: repetitive LLM prompts.

  • Users repeatedly ask the same questions in chatbots and customer support portals.
  • Dashboard widgets and analytics summary cards refresh on every page reload.
  • Automated workflow triggers re-process identical document fragments.
  • Rate limits on fast inference engines (like Groq) are hit unnecessarily by repeated queries.

💰 Return on Investment (ROI) Breakdown

Here is what caching 35% of identical queries looks like for a startup serving 100,000 queries per day:

Metric Without Cache With django-llm-cache Savings / Advantage
API Calls to LLM Provider 100,000 / day 65,000 / day 35,000 fewer API calls daily
Average Response Latency ~1,200 ms 3 ms (Cached Hits) 99.7% latency drop on hits
Monthly Token Cost (GPT-4o) ~$15,000 / mo ~$9,750 / mo $5,250 / month saved ($63,000 / yr)
Rate Limit Headroom High risk of 429 errors Unlocked 35%+ extra throughput Zero throttle downtime
Database/Cache Failures Typically crashes app Fails open to live API Zero downtime guarantee

🏗️ Architecture & Request Lifecycle

sequenceDiagram
    autonumber
    actor Client as User / Frontend
    participant View as Django View / DRF
    participant Cache as django-llm-cache (LLMResponseCache)
    participant Redis as Django Cache Backend (Redis / Memcached)
    participant LLM as LLM Provider (Groq / OpenAI / Gemini)

    Client->>View: POST /api/chat/ {prompt: "Summarize Q3 report"}
    View->>Cache: cache.get(prompt, provider="groq", model="llama-3.3-70b-versatile")
    Cache->>Cache: Generate SHA-256 Key(provider + model + cleaned prompt)
    Cache->>Redis: GET key

    alt Cache Hit (Instant Return)
        Redis-->>Cache: Return cached response payload
        Cache-->>View: Cached response
        View-->>Client: 200 OK (Latency: ~3ms, Cost: $0.00)
    else Cache Miss / First Query
        Redis-->>Cache: None
        Cache-->>View: None
        View->>LLM: Live API call (Inference)
        LLM-->>View: Return LLM response
        View->>Cache: cache.set(prompt, provider, model, response, timeout=300)
        Cache->>Redis: SET key response (TTL 300s)
        View-->>Client: 200 OK (Latency: ~1200ms)
    else Cache Backend Offline / Network Error
        Redis--xCache: ConnectionError / Timeout
        Note over Cache: Caught exception, logged warning.<br/>Fails open without crashing!
        Cache-->>View: None
        View->>LLM: Fallback directly to live API
        LLM-->>View: Return response
        View-->>Client: 200 OK (Zero interruption to end user)
    end

✨ Key Features

  • Deterministic SHA-256 Hashing: Strips prompt whitespace and creates canonical keys combining provider, model_name, prompt content, and optional deterministic hyperparameters (temperature, system_prompt).
  • Resilient Fail-Open Design: If your Redis server crashes, your database runs out of connections, or network lag occurs, django-llm-cache catches the error, logs it via Python's standard logging, and cleanly returns None. Your live user request never crashes.
  • Lazy Django Imports: Safe to import in CLI scripts, background workers, or standalone scripts without throwing AppRegistryNotReady or ImproperlyConfigured. Emits clean, non-fatal warnings when settings are unconfigured.
  • Pluggable Cache Backends: Works with any Django cache backend (django-redis, Memcached, Local Memory, FileSystem, or Database cache). You can specify dedicated cache partitions using the cache_alias parameter.
  • Flexible Expiration: Configurable default TTL (default: 300 seconds) with per-call overrides.

📦 Installation

pip install django-llm-cache

Ensure Django (>=3.2) is configured in your project settings:

# settings.py
CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.redis.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/1",
    }
}

🚀 Quickstart & Code Examples

1. Integration with Groq Python SDK

from groq import Groq
from django_llm_cache import LLMResponseCache

# Initialize cache manager (defaults to 300s expiration)
llm_cache = LLMResponseCache(default_timeout=300)
client = Groq()

def ask_groq(prompt: str) -> str:
    provider = "groq"
    model = "llama-3.3-70b-versatile"

    # 1. Check local cache
    cached_content = llm_cache.get(prompt, provider=provider, model_name=model)
    if cached_content is not None:
        return cached_content

    # 2. Cache miss: Execute live API call
    chat_completion = client.chat.completions.create(
        messages=[{"role": "user", "content": prompt}],
        model=model,
    )
    result_text = chat_completion.choices[0].message.content

    # 3. Store result in cache for future calls
    llm_cache.set(prompt, provider=provider, model_name=model, response=result_text)
    return result_text

2. Integration with OpenAI Python SDK

from openai import OpenAI
from django_llm_cache import LLMResponseCache

llm_cache = LLMResponseCache(default_timeout=600)  # 10 minutes TTL
client = OpenAI()

def ask_openai(prompt: str) -> str:
    provider = "openai"
    model = "gpt-4o"

    # One-line lookup or fetch pattern
    def fetch_from_api():
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
        )
        return response.choices[0].message.content

    # Automatically checks cache; calls fetch_from_api() only on cache miss!
    return llm_cache.get_or_set(
        prompt=prompt,
        provider=provider,
        model_name=model,
        default_callable=fetch_from_api,
    )

3. Integration with Google Gemini SDK

from google import genai
from django_llm_cache import LLMResponseCache

llm_cache = LLMResponseCache(default_timeout=900)
client = genai.Client()

def ask_gemini(prompt: str) -> str:
    provider = "gemini"
    model = "gemini-2.5-flash"

    cached_text = llm_cache.get(prompt, provider, model)
    if cached_text:
        return cached_text

    response = client.models.generate_content(
        model=model,
        contents=prompt,
    )
    llm_cache.set(prompt, provider, model, response.text)
    return response.text

4. Integration in a Django REST Framework (DRF) View

from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status
from django_llm_cache import LLMResponseCache
from groq import Groq

cache = LLMResponseCache(default_timeout=600, key_prefix="customer_chat")
groq_client = Groq()

class AssistantView(APIView):
    def post(self, request):
        user_prompt = request.data.get("prompt", "").strip()
        if not user_prompt:
            return Response({"error": "Prompt is required"}, status=status.HTTP_400_BAD_REQUEST)

        provider = "groq"
        model = "llama-3.3-70b-versatile"

        # Check cache
        cached_response = cache.get(user_prompt, provider, model)
        if cached_response is not None:
            return Response({"answer": cached_response, "source": "cache"})

        # Call live LLM
        completion = groq_client.chat.completions.create(
            messages=[{"role": "user", "content": user_prompt}],
            model=model,
        )
        answer = completion.choices[0].message.content

        # Cache response
        cache.set(user_prompt, provider, model, answer)

        return Response({"answer": answer, "source": "live_api"})

⚙️ Advanced Configuration

Dedicated Redis Cache Alias

If you want to isolate your LLM cache to a specific Redis database or Redis cluster without polluting your session or template cache:

# settings.py
CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.locmem.LocMemCache",
    },
    "llm_redis": {
        "BACKEND": "django.core.cache.backends.redis.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/2",
        "TIMEOUT": None,  # Handled per key by django-llm-cache
    },
}

Then pass the alias to LLMResponseCache:

cache = LLMResponseCache(
    default_timeout=1800,          # 30 minutes
    cache_alias="llm_redis",       # Routes to 'llm_redis' cache in settings.py
    key_prefix="prod_v1",          # Custom key namespace
)

Deterministic Hyperparameter Sensitivity

If responses vary based on sampling parameters like temperature or system_prompt, pass them as keyword arguments into .get(), .set(), or .generate_key():

# Keys generated will differ if temperature or system_prompt change
cached = cache.get(
    prompt="Explain photosynthesis",
    provider="openai",
    model_name="gpt-4o",
    temperature=0.2,
    system_prompt="Explain for a 5-year old",
)

Manual Key Invalidation

# Invalidate a specific prompt entry
cache.delete("Explain photosynthesis", provider="openai", model_name="gpt-4o")

🧪 Running Tests

The test suite covers deterministic key hashing, error handling, lazy configuration checks, and cache routing.

# Clone the repository
git clone https://github.com/Hafiz-alt/django-llm-cache.git
cd django-llm-cache

# Run tests
python -m unittest discover -s tests -v

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.


👤 Author & Support

Metadata

Release files for django-llm-cache 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for django-llm-cache 0.1.0
File Size Uploaded Provenance Actions
django_llm_cache-0.1.0.tar.gz 17.9 kB Without provenance Details for django_llm_cache-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for django-llm-cache 0.1.0
File Interpreter ABI Platform Actions
django_llm_cache-0.1.0-py3-none-any.whl Python 3 none any Details for django_llm_cache-0.1.0-py3-none-any.whl

Total release size: 28.6 kB

Release files / django_llm_cache-0.1.0.tar.gz

Download URL django_llm_cache-0.1.0.tar.gz
Size 17.9 kB
Tags Source
SHA-256 checksum
How to use checksums
e6b137cf5ab29d2535963fd7c8cfb2d0e5ae8a4b77edcbc4c68ec5d730e8ab95
BLAKE2b-256 checksum
How to use checksums
456dc8531b8b9113063a8475536845fa347ed27d3c71f1f8c23f8c1099b4a626
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.10

Release files / django_llm_cache-0.1.0-py3-none-any.whl

Download URL django_llm_cache-0.1.0-py3-none-any.whl
Size 10.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4209693430c0dbf5bcbdc02c70819e1c8a3ff534e93a4402c78379af3948b952
BLAKE2b-256 checksum
How to use checksums
cd00b791c4730e38f79d985fda986fe1867993bb6590d620d304d1759581a43b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.10

Security for django-llm-cache 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page