demystify-platform-cache 0.3.0

Production caching seam for AI workloads (Python mirror): exact-match cache, verified semantic cache, embedding cache, singleflight request coalescing, negative cache, and epoch-based invalidation. In-memory reference default; Redis extra.

@demystify/platform-cache · demystify-platform-cache

Production caching seam for AI workloads — the layer that makes the Demystify suite fast and cost-efficient without ever trading away correctness. Dual-language leaf package: @demystify/platform-cache (TypeScript) and demystify-platform-cache (Python), with an in-memory reference adapter (keyless, offline, deterministic — the default) and a Redis adapter (integration-marked, opt-in).

Built from the production playbook, not the demo playbook:

• Real-world semantic-cache hit rates are 20–40%, and the hard problem is false hits — embedding-close is not meaning-equal ("capital of France" ≈ "capital of Germany" in dense-embedding space).
• A cache amplifies whatever it stores — errors, refusals, and low-groundedness answers must never be admitted.
• TTLs don't know when your data changed — event-driven (epoch) invalidation does.

What's in the box

Primitive	What it does	Guarantee
ExactCache + exactKey	digest of the full canonical request → value	zero false hits by construction
SemanticResponseCache	embed → nearest-neighbour ≥ threshold → verify before serve; admission-gate before store	verified paraphrase hits
TieredCache	exact tier first, then semantic	free/safe hits absorbed before any similarity math
withEmbeddingCache	content-hash → vector decorator over any batch embedder	pure-function cache: can never be wrong
SingleFlight	N concurrent identical misses → 1 upstream call	thundering-herd guard
NegativeCache	short-TTL tombstones for known-empty results	bounded staleness
EpochStore	monotonic epoch per scope, folded into keys	O(1) invalidation on ingest/delete
CacheStats	hit/miss/store/reject events + µUSD saved/spent	measure the false-hit rate, not just the hit rate

Every namespace is tenant-scoped by the caller and every key carries a version segment (embedder/model/schema/epoch), so version swaps invalidate instead of corrupting.

Quickstart (TypeScript)

import {
  createMemoryCacheBackend,
  HashCacheEmbedder,
  SemanticResponseCache,
  TieredCache,
} from "@demystify/platform-cache";

const backend = createMemoryCacheBackend();
const semantic = new SemanticResponseCache<string>({
  embedder: new HashCacheEmbedder(), // deterministic reference; swap a real embedder in prod
  store: backend.semantic,
  admission: (answer) =>
    answer.startsWith("ERROR") ? { admit: false, reason: "error response" } : { admit: true },
});
const cache = new TieredCache<string>({
  exact: backend.exact,
  semantic,
  keyVersion: `epoch:${await backend.epochs.current("tenant:corpus")}`,
});

const hit = await cache.lookup("tenant-1", requestPayload, questionText);
if (!hit) {
  const answer = await expensiveCall();
  await cache.store("tenant-1", requestPayload, questionText, answer, { costMicroUsd: 1200 });
}

Quickstart (Python)

from demystify_platform_cache import (
    HashCacheEmbedder, MemoryCacheBackend, SemanticResponseCache, TieredCache,
)

backend = MemoryCacheBackend()
semantic = SemanticResponseCache(embedder=HashCacheEmbedder(), store=backend.semantic)
cache = TieredCache(exact=backend.exact, semantic=semantic, key_version="epoch:0")

hit = await cache.lookup("tenant-1", request_payload, question_text)

The two guards (why this is safe to run in production)

1. Admission — a quality gate before store: never cache errors, tool calls, abstentions, or low-groundedness answers. The gateway and RAG integrations pass their own policies.
2. Verification — a guard before serve on every semantic candidate. The deterministic reference is content-token overlap (kills the France/Germany class of false hit); real deployments swap a cross-encoder or LLM judge behind the same HitVerifier signature.

Who uses it in the suite

• dmstfy-gateway — exact + verified-semantic completion cache (DMSTFY_GATEWAY_SEMANTIC_CACHE*, OFF by default), singleflight, saved-µUSD accounting.
• dmstfy-rag — embedding cache (ingest + query), retrieval cache and answer cache keyed by corpus epoch (every ingest/delete invalidates instantly), groundedness-gated admission (DMSTFY_RAG_CACHE*, OFF by default).
• dmstfy-cache (module) — make cachebench: the offline proof harness that reports per-tier hit rates, false-hit rate, and µUSD saved, ON vs OFF.

Install

npm add @demystify/platform-cache          # TypeScript
pip install demystify-platform-cache        # Python (redis extra: [redis])

Caches are never the default (CONVENTIONS.md §14.7): every consumer wires them behind an explicit opt-in.

License

MIT © Demystify Systems