orchestra-llm-cache 0.3.1

Reduce AI orchestration costs by 85% with semantic caching

🎵 Orchestra

Unbreakable Agents. 85% Lower Costs. One Line of Code.

---

📺 Watch: How to drop your LLM bill by 85% instantly

---

🚀 The "Zero-Rewrite" AI Framework

Orchestra is a high-performance orchestration layer that adds Semantic Caching, Outage Resilience, and Smart Tool Discovery to LangGraph and LangChain.

It understands the meaning of requests, skipping the LLM entirely when a semantically similar query has already been answered.

graph TD
    User([User Query]) --> Match{Semantic Match?}
    Match -- "92% Similar" --> Cache[(Semantic Cache)]
    Cache --> Return([Instant Response < 0.5s])
    Match -- "New Query" --> CB{Circuit Breaker}
    CB -- "Closed" --> LLM[LLM Provider]
    LLM --> Return
    CB -- "Open" --> Error[Graceful Fallback]

---

⚡ Why Orchestra?

• 📉 Slash OpenAI/Anthropic Bills: Stop paying for the same questions phrased differently.
• 🏎️ Lightning Performance: Sub-second responses for cached "loophole" hits.
• 🛡️ Outage Insurance: Built-in Circuit Breakers keep your app alive even if your LLM provider goes down.
• 🔌 Smart Tool Selection: Automatically filters 100+ tools down to the relevant 5, saving thousands of context tokens.
• 🛠️ Framework Native: Works with LangGraph, LangChain, and any Python LLM client.

---

📦 Installation

pip install orchestra-llm-cache[full]

---

⚡ 1-Line Quick Start

langgraph

from orchestra import enhance

# Wrap your compiled graph. That's it.
# Orchestra now handles Caching, Tracing, and Resilience.
agent = enhance(app.compile())

result = agent.invoke({"query": "What are our Q3 goals?"})

langchain

from orchestra import enhance

# Works with any Runnable or Chain
chain = enhance(my_rag_chain)

result = chain.invoke("Explain the merger")

---

🛡️ Enterprise Resilience (Circuit Breaker)

Stop cascading failures. If your LLM provider starts timing out, Orchestra opens the circuit, failing fast and serving cached results until the provider recovers.

from orchestra import OrchestraConfig

config = OrchestraConfig(
    enable_circuit_breaker=True,
    circuit_breaker_threshold=5, 
    circuit_breaker_timeout=60.0
)
agent = enhance(graph, config=config)

---

🔌 Smart Tool Discovery (MCP)

Are you loading 50+ tools into Claude? You're burning money. Orchestra indexes your MCP tools semantically and only injects the ones relevant to the current user query.

config = OrchestraConfig(
    enable_tool_search=True,
    tool_search_top_k=5  # Only show the best 5 tools
)

---

💾 Production Backends

Backend	Best For	Description
SQLite	Development	Local, file-based, zero-config.
PostgreSQL	Tracing	Centralized telemetry for large teams.
Redis Stack	Distributed	Global semantic cache shared across pods.

---

🎥 CLI & Observability

Debug your agents with trace logs.

# View recent executions
python -m orchestra.cli trace ls

# Compare two strings semantically
python -m orchestra.cli eval "Hello" "Hi there"

---

❓ FAQ

Q: How accurate is the semantic matching?
A: By default, we use a 0.92 cosine similarity threshold. It's high enough to ensure accuracy but loose enough to catch "How's the weather?" vs "What's the weather like?".

Q: Does this work with exact matches?
A: Yes. If a query is identical, it hits the cache instantly. If it's a paraphrase, we use vector embeddings.

---

Start building unbreakable, affordable AI agents today. 🌟 Give us a star on GitHub!