Waggle

Repository: Waggle-SemCache

Built by Abhigyan Shekhar

Local semantic caching for LLMs, RAG pipelines, and AI agents.

Reuse expensive AI computations when requests are semantically equivalent—without Redis, a vector database, an API key, or a cloud service.

• Local-first and SQLite-backed
• Exact caching in the dependency-free core; semantic matching through pluggable providers
• Context-aware validation, TTLs, metadata, and invalidation
• Sync and async APIs with duplicate-work coalescing
• Atomic LRU/FIFO storage budgets and optional cross-process single-flight
• Inspectable match decisions
• No telemetry

Quick start

pip install waggle-cache

from waggle import Waggle

cache = Waggle("./cache.db")
answer = cache.get_or_compute(
    key="Explain gradient descent",
    compute=lambda: expensive_llm_call(),
    namespace="education",
)

The base package uses only the Python standard library and starts in exact-only mode. It does not install Torch, Transformers, an SDK, or a model. Exact entries do not contain vectors and are not reported as missing from the vector index.

For local semantic matching, install the optional provider and configure it explicitly:

pip install "waggle-cache[local]"

from waggle import Waggle
from waggle.embeddings import LocalEmbeddingProvider

cache = Waggle(
    "./cache.db",
    embedding_provider=LocalEmbeddingProvider(),
)

LocalEmbeddingProvider lazily downloads all-MiniLM-L6-v2 on its first semantic operation and then runs locally on CPU. You can instead pass any object implementing model_id and embed(list[str]) to use an embedding API, another local runtime, or an application-owned model.

The bundled local provider is pinned to MiniLM revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41 so persisted vectors have reproducible identity. Custom unpinned models are explicitly stored with an @unversioned identity.

When semantic is omitted, Waggle enables semantic lookup only when an embedding provider is configured. Passing semantic=True without a provider raises MissingEmbeddingProviderError instead of silently pretending to perform semantic matching.

Bounded local storage

Production caches should not grow forever. Configure entry and/or logical-byte budgets directly on the cache:

cache = Waggle(
    "./cache.db",
    max_entries=10_000,
    max_size_mb=500,
    eviction="lru",  # or "fifo"
)

Every insertion purges expired and invalidated rows, commits the new entries, and evicts older live entries inside one short SQLite transaction. Atomic batches larger than the configured capacity raise CacheCapacityError without changing the cache. stats() reports logical storage_bytes and evictions; call vacuum() explicitly when physical SQLite page reclamation is required.

Safe semantic reuse

Nearest does not mean reusable. Waggle first tries a normalized exact lookup, then embeds on an exact miss. Semantic candidates must pass every configured gate:

1. same namespace and embedding model;
2. similarity threshold;
3. unexpired and not invalidated;
4. matching context fingerprint;
5. matching metadata filter; and
6. the optional verifier.

result = cache.get(
    "Summarize this document",
    namespace="docs",
    context={"document_version": "v13", "model": "my-model"},
    metadata_filter={"tenant": "acme", "language": "en"},
    debug=True,
)

print(result.hit, result.match_type, result.similarity)
for candidate in result.candidates:
    print(candidate.similarity, candidate.accepted, candidate.reason)

Context is canonicalized and SHA-256 fingerprinted; raw context is not stored. Configure context_fields or ignore_context_fields on Waggle to define compatibility. A missing context and a supplied context are intentionally different.

Application cache policies

Similarity cannot determine whether a workload is cacheable or which business fields must remain identical. Supply a policy to bypass unsafe requests and extract structured reuse invariants before any cache lookup:

from waggle import CachePolicyDecision, CacheRequest, Waggle
from waggle.embeddings import LocalEmbeddingProvider


def support_policy(request: CacheRequest) -> CachePolicyDecision:
    context = request.context or {}
    if context.get("personalized"):
        return CachePolicyDecision.bypass("personalized request")
    return CachePolicyDecision(
        invariants={
            "tenant": context.get("tenant"),
            "knowledge_base_version": context.get("knowledge_base_version"),
            "operation": context.get("operation"),
        },
        policy_version="support-v1",
    )


cache = Waggle(
    "./cache.db",
    embedding_provider=LocalEmbeddingProvider(),
    policy=support_policy,
    context_fields=("model",),
)

A bypassed get_or_compute runs the function without reading or writing the cache and returns match_type="bypass" when return_result=True. Invariants and the policy version are canonicalized into a separate SHA-256 fingerprint; raw invariant values are not persisted. A changed invariant is reported as invariant mismatch. Increment policy_version whenever a policy's compatibility meaning changes.

Policies may also be supplied per call or decorator with policy=.... An explicit set or set_many rejected by policy raises before embedding or writing; a mixed rejected batch remains atomic.

OpenAI-compatible integration

Bring the provider client configured by your application; the base Waggle installation adds no OpenAI SDK dependency:

from openai import OpenAI

from waggle import Waggle
from waggle.embeddings import LocalEmbeddingProvider
from waggle.integrations.openai import wrap_openai

client = OpenAI()
client = wrap_openai(
    client,
    Waggle("./cache.db", embedding_provider=LocalEmbeddingProvider()),
)

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[{"role": "user", "content": "Explain gradient descent"}],
    temperature=0,
)

Existing call sites keep the same client.chat.completions.create(...) and client.responses.create(...) paths and receive native SDK objects on misses and hits. The wrapper fingerprints the model, conversation history, tools, response format, and generation parameters while using the final text user input for semantic matching. Multimodal requests fall back to exact hashed keys and streaming passes through uncached. See the OpenAI-compatible integration guide.

LangChain integration

Register Waggle through LangChain's native global cache hook; existing model calls do not change:

pip install 'waggle-cache[langchain,local]'

from langchain_core.globals import set_llm_cache

from waggle import Waggle
from waggle.embeddings import LocalEmbeddingProvider
from waggle.integrations.langchain import WaggleLangChainCache

cache = Waggle("./cache.db", embedding_provider=LocalEmbeddingProvider())
set_llm_cache(WaggleLangChainCache(cache))
answer = model.invoke("Explain gradient descent")

The adapter implements BaseCache sync and async methods, preserves native Generation and ChatGeneration values, and fingerprints the complete LangChain model configuration as a strict reuse boundary. It also accepts per-integration namespaces, TTLs, thresholds, exact-only mode, and cache policies. See the LangChain integration guide.

LiteLLM integration

Enable Waggle once, then keep existing LiteLLM completion calls unchanged:

pip install 'waggle-cache[litellm,local]'

import litellm

from waggle import Waggle
from waggle.embeddings import LocalEmbeddingProvider
from waggle.integrations.litellm import enable_litellm

cache = Waggle("./cache.db", embedding_provider=LocalEmbeddingProvider())
enable_litellm(cache)
response = litellm.completion(model="openai/gpt-4.1-mini", messages=[...])

The native backend supports completion() and acompletion(), preserves LiteLLM ModelResponse objects, and isolates provider/model parameters and conversation history while matching the final text user message semantically. See the LiteLLM integration guide.

API

cache.set(key, value, namespace="support", ttl=3600, context={...}, metadata={...})
cache.set_many([("first request", value1), ("second request", value2)], namespace="support")
result = cache.get(key, namespace="support", threshold=0.94)
value = cache.get_or_compute(key, compute=callable, namespace="support")

cache.invalidate(entry_id=result.entry_id)
cache.invalidate(key=key, namespace="support")
cache.invalidate_where({"document_version": "v1"}, namespace="support")
cache.clear(namespace="support")
cache.clear_expired()
cache.stats()
cache.vacuum()

get() always returns a CacheResult. get_or_compute() returns the cached/computed value by default; use return_result=True for provenance. Values and metadata must be JSON serializable. Waggle never uses pickle.

Decorators

@cache.semantic(
    namespace="support",
    ttl=3600,
    threshold=0.85,
    key=lambda question, user_id: question,
    context=lambda question, user_id: {"user_id": user_id},
)
def answer(question: str, user_id: str) -> str:
    return call_llm(question)

Use @cache.exact(...) when only identical normalized inputs may share results. Both decorators support async functions.

Async

answer = await cache.aget_or_compute(
    key=query,
    compute=lambda: ask_llm(query),
    namespace="support",
)

Blocking SQLite and embedding work runs off the event loop. The awaited computation itself stays async. Concurrent identical misses are coalesced per process without holding a global lock during the slow call.

For Gunicorn, multiprocessing, and other multi-worker deployments, opt into SQLite-backed leases:

cache = Waggle(
    "./cache.db",
    cross_process_singleflight=True,
    lease_ttl=60,
)

Lease acquisition and renewal use short transactions; the expensive computation runs without a SQLite lock. Waiting workers recheck the cache, crashed owners are replaced after expiry, and a heartbeat protects long-running computations. Only identical scoped request keys are coalesced—semantic neighbors are never merged in flight.

Large local indexes

The default exact cosine backend is dependency-free and deterministic. For large namespaces, install the optional persistent HNSW backend:

pip install 'waggle-cache[ann,local]'

cache = Waggle(
    "./cache.db",
    embedding_provider=LocalEmbeddingProvider(),
    index_backend="hnsw",
)

HNSW files are stored beside the SQLite database, separately for each namespace and embedding model. SQLite remains the source of truth. Every HNSW file carries the corresponding SQLite generation and is rebuilt automatically when missing, stale, corrupt, or incompatible. Candidate similarities are recalculated exactly. Hard compatibility filters run before ranking, and HNSW retrieval widens adaptively until a valid candidate is found, the scope is exhausted, or max_semantic_candidates is reached.

Exact embedding cache vs semantic response cache

An embedding computation cache maps an exact text to its previous vector. A semantic response cache maps a sufficiently equivalent request to a previous computation. For an embedding cache, use semantic=False or @cache.exact; never semantically reuse an embedding for different text.

CLI

waggle -d ./cache.db stats
waggle -d ./cache.db namespaces
waggle -d ./cache.db inspect "How does binary search work?" --namespace docs
waggle -d ./cache.db clear --namespace support
waggle -d ./cache.db clear-expired
waggle -d ./cache.db vacuum
waggle -d ./cache.db rebuild-index
waggle -d ./cache.db doctor
waggle -d ./cache.db --index-backend hnsw rebuild-index
waggle -d ./cache.db benchmark-thresholds benchmarks/paraphrases.json
waggle -d ./cache.db evaluate-policies benchmarks/policy_cases.json --thresholds 0.80 0.85 0.90

Storage, privacy, and recovery

The configured SQLite file contains keys, JSON values, metadata, fingerprints, timestamps, and float32 vectors. WAL mode supports concurrent readers and safe transactions. No data or telemetry leaves the machine, and secrets should be represented by fingerprints or version identifiers—not stored as metadata.

SQLite is the source of truth. The dependency-free cosine index is rebuilt from stored vectors and can later be replaced behind the same boundary by an approximate index. doctor runs database integrity and vector compatibility checks.

Benchmarks

python benchmarks/benchmark.py --entries 1000
python benchmarks/thresholds.py benchmarks/paraphrases.json

The scripts report measurements from the current machine. This README intentionally contains no invented performance numbers. See benchmark methodology, threshold and exact-index calibration, transactional batching with persistent HNSW, and the multi-domain cache-policy evaluation.

The default threshold is 0.85, not a universal optimum. The generic verifier is intentionally too small to encode domain correctness. Run evaluate-policies on your own workload and put cacheability rules, tenant boundaries, versions, languages, operations, and other business constraints in application policy or structured invariants.

Development

python -m pip install -e '.[dev]'
pytest
ruff check .

See architecture and migrations. Waggle is built by Abhigyan Shekhar. This Python cache library is standalone from Waggle MCP: it has no MCP runtime dependency and does not reuse the memory engine's knowledge-graph architecture or source code.

License

Apache-2.0.