thriftlm 0.1.0

Semantic caching layer for LLM applications — stop paying for the same call twice.

ThriftLM

Stop paying for the same LLM call twice just because users phrased it differently.

ThriftLM is a drop-in semantic caching layer for LLM applications. It embeds each query with SBERT, checks a vector store for semantically similar past queries, and returns the cached response when similarity is above threshold — no LLM call required.

Install

pip install thriftlm

Quickstart

from semantic_cache import SemanticCache

cache = SemanticCache(api_key="sc_xxx")
response = cache.get_or_call(query, llm_fn=my_langchain_chain.invoke)

That's it. No architecture changes — just wrap the existing LLM call.

Self-hosting

cp .env.example .env  # fill in SUPABASE_URL, SUPABASE_KEY, REDIS_URL
docker compose up

Set THRIFTLM_URL=http://localhost:8000 in your app to point at the local instance.

Docs

thriftlm.dev/docs

License

MIT