intellicache 0.1.0

An intelligent LLM caching library that combines semantic similarity, configurable judgment, and persistent vector storage to safely reuse responses across equivalent queries.

IntelliCache : A Library for Creating Matching Cache for LLM Queries

What is IntelliCache

Various large language models (LLMs) boast incredible versatility, enabling the development of a wide range of applications. However, as your application grows in popularity and encounters higher traffic levels, the expenses related to LLM API calls can become substantial. While semantic caching systems (like gptcache) exist to build a semantic cache for storing LLM responses, Intellicache goes beyond semantic caching by handling cases semantic matching may mislabel through a small LLM judge, boosting both caching accuracy and precision in a cost-effective way.

Why IntelliCache?

Exact caches miss differently worded versions of the same request. Pure similarity caches (like gptcache) can make the opposite mistake and reuse an answer for a query that only looks related.

IntelliCache combines:

• Exact query detection
• Configurable semantic matching
• Configurable LLM judgment for ambiguous candidates
• Custom embedding and judge provider interfaces

[!WARNING] IntelliCache is alpha software. Its API and on-disk format may change before a stable release. Review the current limitations before using it with production data

Installation

IntelliCache requires Python 3.11 or newer.

The recommended setup uses local Sentence Transformers embeddings and an OpenAI judge for ambiguous matches:

pip install "intellicache[sentence-transformers,openai]"

Other installation options:

# OpenAI embeddings and judge
pip install "intellicache[openai]"

# Local embeddings without an LLM judge
pip install "intellicache[sentence-transformers]"

# Anthropic Claude judge
pip install "intellicache[anthropic]"

# Local Ollama embeddings and judge
pip install "intellicache[ollama]"

# Install multiple integrations together
pip install "intellicache[anthropic,ollama]"

# Custom providers only
pip install intellicache

Optional integrations are loaded only when used. Importing IntelliCache does not load any provider SDK, and requesting a missing integration produces an error showing which extra to install.

Built-in providers

Embedding providers:

• OpenAIEmbedding
• SentenceTransformerEmbedding
• OllamaEmbedding

Judge providers:

• OpenAIJudge
• AnthropicJudge
• OllamaJudge

Providers can be mixed. For example, use OpenAIEmbedding() with AnthropicJudge(). Hosted providers read their conventional environment variables: OPENAI_API_KEY or ANTHROPIC_API_KEY. Ollama defaults to http://localhost:11434; its models must already be available on that server.

Quick start

Set your OpenAI API key before running the example:

export OPENAI_API_KEY="your-api-key"

In PowerShell, use $env:OPENAI_API_KEY="your-api-key" instead.

You also can make a .env file containing OPENAI_API_KEY="your-api-key" and import it using dotenv() from python-dotenv

OpenAI API usage:

import asyncio

from intellicache import IntelliCache
from intellicache.embedding import OpenAIEmbedding
from intellicache.judge_evaluator import OpenAIJudge


async def generate_response(query: str) -> str:
    # Replace this with your LLM, RAG pipeline, or API call.
    # Any configuration works so long as it accepts a query
    # str and returns a response str
    return f"Generated response for: {query}"


async def main() -> None:
    cache = IntelliCache(
        embedding_provider=OpenAIEmbedding(model_name="text-embedding-3-small"),
        judge_provider=OpenAIJudge(
            model_name="gpt-5-nano",
            reasoning_effort="low",
            max_output_tokens=1024,
        ),
        cache_dir=".intellicache_test",
        use_judge=True,
        base_threshold=0.97,
        judge_threshold=0.92,
        judge_k=10,
    )

    try:
        # Searches for query in cache, if none exists
        # it generates new response and caches it
        result = await cache.get_or_create(
            query="How many seasons are there in a year?",
            response_factory=generate_response,
        )

        print("Response:", result.response)
        print("Cache hit:", result.hit)
        print("Decision:", result.decision_path)

        # Second search of similar query will likely result in a
        # 'hit' and return the cached response rather than generating
        # a new response
        result = await cache.get_or_create(
            query="How many seasons in a year?",
            response_factory=generate_response,
        )

        print("Response:", result.response)
        print("Cache hit:", result.hit)
        print("Decision:", result.decision_path)

    finally:
        #Needed step to save cache between runs
        cache.close()


asyncio.run(main())

The first query is not yet in the empty cache, so IntelliCache calls generate_response and stores its result. The second query expresses the same request differently. IntelliCache embeds it, compares it with the cached query, and can reuse the first response instead of calling generate_response again. The printed hit and decision_path values show whether reuse occurred and which matching stage made the decision.

get_or_create() is the main cache-aside interface: it searches first, calls response_factory(query) only on a miss, stores the generated response, and returns both the response and information about the cache decision. Because the example uses the persistent .intellicache_test directory, later runs can reuse entries created by earlier runs. Delete that directory when you want to start again with an empty cache.

A similarity at or above base_threshold is accepted directly. A candidate between judge_threshold and base_threshold is sent to the OpenAI judge, which decides whether its cached response is safe to reuse. A lower-scoring query is treated as a miss without a judge call. OpenAI embedding requests and judge requests can both add provider cost and latency, so validate the thresholds against representative data from your application.

Core API

get_or_create()

Use get_or_create() for the normal cache-aside workflow:

async def generate_response(query: str) -> str:
    #Example client, change this for your LLM/RAG/Response system
    response = await my_client.generate(query)
    return response


result = await cache.get_or_create(
    query="How do I reset my password?",
    response_factory=generate_response,
)

The callable runs only on a cache miss. If it raises an exception, no new response is stored.

When a semantic hit occurs, IntelliCache returns the existing response and stores the submitted query as another variant of that response set. This can improve recognition of future phrasings.

search()

Search without calling a generator or inserting a query:

result = await cache.search("Where can I change my password?")

if result.hit:
    print(result.response)
else:
    print("Cache miss")

put_response()

Store a query and a response you already have:

result = await cache.put_response(
    query="How do I reset my password?",
    response="Open account settings and select Reset Password.",
)

If the identical query already exists, the response is reused or updated. A merely similar query does not overwrite an existing response set.

put_variant()

Associate a new phrasing with an existing response:

result = await cache.put_variant(
    query="Where can I update my password?",
)

This method requires a reusable match. It raises KeyError if no suitable response set is found.

Lifecycle and statistics

print(cache.stats())
# {"responses": 1, "queries": 2, "vectors": 2}

cache.save()
cache.close()

close() saves the FAISS index before closing SQLite. Use try/finally when the cache does not live for the entire process.

Cache results

Public operations return a CacheResult:

Field	Meaning
hit	Whether an existing response was reused
query	The submitted query
response	The cached or newly created response
set_id	Identifier of the associated response set
matched_query	Existing query responsible for the match
score	Similarity score, when applicable
decision_path	How the result was selected

Common decision paths include:

• miss
• new_cluster
• exact_duplicate
• updated_response
• base_threshold
• judge

Matching policy

With the default vector configuration, FAISS normalizes vectors and performs inner-product search. For normalized vectors, inner product is equivalent to cosine similarity, so higher scores represent more similar queries.

The matching policy is:

1. Retrieve the nearest stored queries from FAISS.
2. Reuse an identical best query as an exact duplicate.
3. Reuse the best candidate when its score meets base_threshold.
4. If enabled, call the judge when the best score is below base_threshold but at least judge_threshold.
5. Return a miss if no candidate is accepted.

Thresholds are model- and dataset-dependent. Benchmark them against your own queries, with particular attention to false-positive cache hits.

Configuration

Option	Default	Purpose
embedding_provider	required	Produces query embeddings
judge_provider	None	Optionally evaluates ambiguous candidates
use_judge	False	Enables judge evaluation
use_adv_embedding	False	Enables the optional secondary embedding check
adv_embedding_provider	None	Provider used for that secondary check
cache_dir	"cache_storage"	Persistent cache directory; use None for an in-memory cache
base_threshold	0.97	Minimum score for direct semantic reuse
judge_threshold	0.92	Minimum score before consulting the judge
judge_k	10	Number of candidates supplied to the judge
normalize_vectors	True	L2-normalizes vectors before FAISS search
full_storage_validation	True	Validates SQLite and FAISS IDs at startup

When use_judge=True, a judge provider is required.

Pass cache_dir=None for an explicitly non-persistent cache. In-memory caches do not create SQLite, FAISS, or configuration files and are discarded by close() or when the process exits.

Persistence and compatibility

Each cache directory contains:

.intellicache/
├── cache.sqlite
├── faiss.index
└── cache_config.json

cache_config.json records the embedding identity, dimension, vector normalization, search metric, and storage schema version. IntelliCache rejects an incompatible configuration rather than comparing vectors produced by different embedding spaces.

Embedding providers expose a JSON-compatible cache_identity. Identity fields should include settings that affect generated vectors, but never secrets such as API keys.

If SQLite and FAISS contain different vector IDs, IntelliCache rebuilds FAISS from the embeddings stored in SQLite. A populated legacy cache without cache_config.json is rejected because its embedding configuration cannot be verified.

Custom providers

Custom embedding provider

An embedding provider supplies a dimension, stable cache identity, and embedding methods:

from collections.abc import Sequence
from typing import Any

import numpy as np


class MyEmbeddingProvider:
    @property
    def dimension(self) -> int:
        return 384

    @property
    def cache_identity(self) -> dict[str, Any]:
        return {
            "provider": "my-company",
            "model": "support-embeddings-v1",
            "normalize_embeddings": True,
        }

    def embed_one(self, text: str) -> np.ndarray:
        ...

    def embed_many(
        self,
        texts: Sequence[str],
        show_progress_bar: bool = False,
    ) -> np.ndarray:
        ...

The identity must contain only JSON-compatible values and must change whenever the provider begins producing an incompatible vector space.

Pass the custom provider directly to IntelliCache:

cache = IntelliCache(
    embedding_provider=MyEmbeddingProvider(),
)

Adding a Custom LLM-backed judge

Authors of LLM-backed integrations can subclass BaseStructuredJudge instead of rebuilding the orchestration layer. It provides prompt construction, candidate-index parsing, retries, rate limiting, concurrent judge_many() execution, default fallback behavior, and token-usage accounting. A provider adapter only needs to initialize its client and implement one request method:

from typing import Any

from intellicache.judge_evaluator import (
    BaseStructuredJudge,
    JudgeModelResponse,
)


class MyLLMJudge(BaseStructuredJudge):
    def __init__(
        self,
        client: Any,
        model_name: str = "my-fast-classifier",
        *,
        match_criteria: str | None = None,
    ) -> None:
        self.client = client
        self.model_name = model_name
        super().__init__(
            match_criteria=match_criteria,
            max_retries=3,
            default_judgement=None,
            # Add the provider's transient API exception classes here.
            retryable_exceptions=(),
        )

    async def _request_judgement(
        self,
        prompt: str,
        *,
        num_candidates: int,
    ) -> JudgeModelResponse:
        response = await self.client.generate(
            model=self.model_name,
            prompt=prompt,
            # Configure the provider to return JSON when supported.
        )
        return JudgeModelResponse(
            output_text=response.text,
            usage=getattr(response, "usage", None),
        )

The model output must be a JSON object with exactly one decision field:

{"match_index": 0}

Use null when no candidate matches:

{"match_index": null}

The base class validates the result against num_candidates. Provider usage objects with input_tokens, output_tokens, and total_tokens attributes are recorded automatically. Override _record_usage() when an SDK uses different field names. Pass a provider rate limiter to super().__init__() when the service has request limits. Local providers can omit rate_limiter to use the built-in no-op limiter.

cache = IntelliCache(
    embedding_provider=MyEmbeddingProvider(),
    judge_provider=MyJudgeProvider(),
    use_judge=True,
    cache_dir=".intellicache",
)

Custom match criteria

Every built-in LLM judge accepts match_criteria. The text defines what the judge considers safe to reuse:

[!CAUTION] Custom LLM judging is experimental and does not include a complete prompt- injection defense. The judge receives query and candidate-query text, not cached response bodies, which limits direct data exposure; however, hostile query text could still influence a match decision and cause an unintended cached response to be returned. Do not treat the judge as an authorization boundary, and validate it against the kinds of untrusted input your application accepts.

from intellicache.judge_evaluator import OpenAIJudge

support_criteria = """
Match only when one response can answer both queries completely and without
modification.

For account-support requests, the product, requested operation, account state,
platform, and error condition must all be the same.

Reject queries when either one adds or removes a requested step, changes the
product or platform, refers to a different error, or requires an additional
assumption. Shared topic or vocabulary alone is not sufficient.
""".strip()

judge = OpenAIJudge(
    model_name="gpt-5-nano",
    match_criteria=support_criteria,
)

AnthropicJudge and OllamaJudge accept the same argument. Passing None uses IntelliCache's strict default criteria. A custom value replaces the default criteria, so include every rule your application depends on rather than supplying only a short amendment.

Criteria should describe observable equivalence rather than tell the model to be generally "careful." State what details must match, list common reasons to reject, and explicitly require None when uncertain. Validate changes with representative labeled queries: stricter criteria generally reduce false positives but can reduce recall, while looser criteria do the reverse.

The criteria apply only to candidates in the judge band: from judge_threshold up to, but not including, base_threshold. Similarities at or above base_threshold are accepted without a judge, and similarities below judge_threshold miss without one. Tune the thresholds and criteria together.

Benchmark results

This run used local sentence-transformers/all-MiniLM-L6-v2 embeddings and a gpt-5-nano judge with low reasoning, a 0.98 base threshold, and a 0.92 judge threshold.

IntelliCache and GPTCache processed the same 20,000 labeled queries from Yahoo Answers in the same order, including 908 repeat-query opportunities: (used data is accessible at: https://huggingface.co/datasets/Cytadell/yahoo_answers_matches)

System	Correct reuse	Incorrect reuse	Hit precision
IntelliCache judge (0.98/0.92)	467	15	96.89%
GPTCache (0.92)	476	67	87.66%
GPTCache (0.97)	283	7	97.59%

IntelliCache remained within one percentage point of GPTCache 0.92 correct- repeat coverage while reducing incorrect reuse by 77.6%. Against high- precision GPTCache 0.97, it recovered 65.0% more correct hits at similar precision. The judge was invoked for only 1.60% of requests.

Ordered-workload judge cost

The judge used 163,446 input tokens and 99,327 output tokens across 320 calls. At GPT-5 Nano's listed price of $0.05 per million input tokens and $0.40 per million output tokens, the complete 20,000-query run cost approximately $0.048, or $0.0000024 per workload request. The judge directly supplied 225 correct reuses, making its measured cost approximately $0.00021 per direct judge-path correct reuse. Local embedding compute and answer-generation costs are not included. See GPT-5 Nano pricing. For a price-only comparison, applying GPT-5 Mini's $0.25/M input and $2/M output rates to the same recorded token counts would cost approximately $0.24, or 5x as much. An actual Mini rerun could use a different number of tokens and produce different decisions. See GPT-5 Mini pricing.

RAG does not have a universal token cost because retrieved context, system instructions, and answer length vary. For scale, a hypothetical RAG answer using 2,000 input tokens and 300 output tokens would cost:

Answer model	Approximate cost per generated answer	Compared with $0.00021 judge cost per directly recovered hit
GPT-5 Nano	$0.00022	1.0x
GPT-5 Mini	$0.0011	5.2x
GPT-5.6 Luna	$0.0038	18x
GPT-5.6 Terra	$0.0095	45x
GPT-5.6 Sol	$0.0190	89x

This example uses OpenAI's listed model prices as of August 2026. The assumed 300 output tokens are total billed output tokens, including internal reasoning tokens. If a request produces 300 visible tokens plus additional reasoning tokens, its real cost will be higher than this table. It is a transparent break-even illustration, not a claim that every RAG request contains exactly 2,300 billed tokens. On very small Nano generations, judge savings may not cover the judge by themselves; the economics become substantially stronger as the avoided answer model, reasoning effort, or RAG context becomes more expensive.

These results demonstrate the precision/coverage tradeoff on this labeled workload; they are not a claim that every application will produce the same numbers.

Adversarial PAWS pair test

A separate 677-pair PAWS test from Quora Question Pairs measured resistance to highly similar queries whose meaning may change when words or entities are swapped. Both systems used text-embedding-3-small and the same 0.95 candidate threshold. IntelliCache used a 0.9999 direct-hit threshold and sent ambiguous candidates to a low-reasoning gpt-5-nano judge.

System	Accuracy	Hit precision	Recall	False-positive rate	False positives
IntelliCache judge	82.57%	67.30%	74.35%	14.20%	69
GPTCache	41.21%	31.94%	95.81%	80.25%	390

IntelliCache reduced false positives by 82.3% and improved accuracy by 41.36 percentage points. In the paired comparison, IntelliCache alone was correct on 321 examples while GPTCache alone was correct on 41. The tradeoff was lower recall and higher latency. Because PAWS is intentionally adversarial, the judge ran on 572 of 677 pairs (84.5%); that call rate should not be treated as representative production traffic.

Current limitations

• A cache directory represents one trusted application context. Do not share it across applications, system prompts, knowledge bases, or users that require data isolation.
• Queries, responses, metadata, and embeddings are stored locally without application-level encryption.
• Entries do not expire automatically, and there is not yet a public deletion or eviction API.
• Concurrent misses for the same query are not deduplicated and may call the generator more than once.
• Cross-process and multithreaded access guarantees are not yet defined.
• Corrupted FAISS files may require removing faiss.index so it can be rebuilt from SQLite.
• The advanced embedding path is experimental.
• The public API and storage schema may change during the alpha period.

Development and testing

for local development:

git clone https://github.com/Cytadell/intellicache.git
cd IntelliCache
pip install -e ".[openai,sentence-transformers,test]"

Run the deterministic offline suite:

python -m pytest -q

The default tests use fake providers and do not make external API calls.

Slow provider tests are skipped unless explicitly enabled:

python -m pytest --runslow

Slow tests may download models, make network requests, use API credentials, and incur provider charges.

Build and validate distribution artifacts:

python -m build
python -m twine check dist/*

License

IntelliCache is available under the MIT License.