ferrocache

Python client, SDK middleware, and framework integrations for ferrocache — a distributed semantic cache for LLM applications written in Rust.

Install

pip install ferrocache                    # base client (zero deps)
pip install ferrocache[openai]            # + OpenAI middleware
pip install ferrocache[anthropic]         # + Anthropic middleware
pip install ferrocache[langchain]         # + LangChain cache backend
pip install ferrocache[llamaindex]        # + LlamaIndex LLM wrapper
pip install ferrocache[mcp]               # + MCP server for Claude Desktop
pip install ferrocache[all]               # everything

Quick start

from ferrocache import FerrocacheClient

client = FerrocacheClient("http://localhost:3000")
client.insert(embedding=[0.1, 0.2, 0.3], response="cached answer", query_text="my query")
hit = client.query(embedding=[0.1, 0.2, 0.3], threshold=0.92)

SDK middleware (one-line integration)

from openai import OpenAI
from ferrocache.middleware import wrap_openai

client = wrap_openai(OpenAI())  # all chat completions now check cache first

from anthropic import Anthropic
from ferrocache.middleware import wrap_anthropic

client = wrap_anthropic(Anthropic())

Framework integration

# LangChain
from langchain.globals import set_llm_cache
from ferrocache.langchain import FerrocacheCache
set_llm_cache(FerrocacheCache())

# LlamaIndex
from llama_index.llms.openai import OpenAI
from ferrocache.llamaindex import FerrocacheLLM
llm = FerrocacheLLM(inner=OpenAI(model="gpt-4o-mini"))

MCP server (Claude Desktop / Claude Code)

pip install ferrocache[mcp]
python3 -m ferrocache.mcp_server

Three tools — semantic_cache_lookup, semantic_cache_store, cache_status — accept text and embed locally before talking to the cache. See the setup guide for Claude Desktop / Claude Code config.

Configuration

All wrappers default to fail_open=True — a cache outage falls through to the real call rather than crashing.

See the main repo for the Rust server, benchmarks, and full architecture docs.