langchain-cachly

Cachly semantic cache for LangChain — fleet-wide LLM response caching by meaning, not by exact string match. Zero extra dependencies.

The problem

Your users ask the same question dozens of different ways. Without semantic caching, every rephrasing hits your LLM and costs money. RedisCache only catches exact duplicates. RedisSemanticCache requires a local embedding model per instance.

langchain-cachly uses server-side similarity search — one managed service, shared across your whole fleet.

Quick start

pip install langchain-cachly

import langchain
from langchain_cachly import CachlySemanticCache

langchain.llm_cache = CachlySemanticCache(
    vector_url="https://api.cachly.dev/v1/sem/YOUR_VECTOR_TOKEN",
    threshold=0.92,   # cosine similarity 0–1; 0.92 recommended
    ttl=86400,        # seconds; default 24 h
)

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o")

llm.predict("What is semantic caching?")           # → LLM called, answer cached
llm.predict("Can you explain semantic caching?")   # → cache HIT, $0 LLM cost

Get your CACHLY_VECTOR_URL from cachly.dev — free tier available.

Why CachlySemanticCache?

Configuration

Async support

CachlySemanticCache implements the full BaseCache interface including alookup, aupdate, and aclear via thread-pool delegation — compatible with langchain's async chains out of the box.

Fail-open design

Network errors and HTTP failures return None (cache miss) and are logged at WARNING level. The cache never raises in the hot path — your LLM calls always complete even if cachly is temporarily unreachable.

Links

• cachly.dev — Free signup, dashboard, pricing
• Docs — Full integration docs
• GitHub