Binary Semantic Cache

Cut LLM costs by 50-90% with sub-millisecond latency.

A high-performance, enterprise-grade semantic cache for OpenAI and local LLMs. Powered by a Rust core for maximum speed, memory efficiency, and fast startup.

⚡ Why Use This?

Most semantic caches are slow (Python-only), heavy (require VectorDB), or complex (require Redis).

Benchmark Source: benchmarks/results/cache_e2e_bench.json (Intel i7, 100k entries).

🚀 Quick Start

1. Installation

Prerequisites: Python 3.10+. Rust is only needed for source builds.

# Option A: From PyPI (Recommended)
pip install "binary-semantic-cache[openai]"

# Option B: From Source (Development)
git clone https://github.com/matte1782/binary_semantic_cache.git
cd binary_semantic_cache
pip install maturin
maturin develop --release --extras openai

2. Choose Your Backend

A. OpenAI (Production)

Best for production apps. Includes automatic rate limiting and cost tracking.

import os
from binary_semantic_cache import BinarySemanticCache, BinaryEncoder
from binary_semantic_cache.embeddings.openai_backend import OpenAIEmbeddingBackend

# 1. Setup Backend (Tier 1 rate limits default)
os.environ["OPENAI_API_KEY"] = "sk-..."
backend = OpenAIEmbeddingBackend(model="text-embedding-3-small")

# 2. Initialize Cache (1536 dimensions for OpenAI)
encoder = BinaryEncoder(embedding_dim=1536, code_bits=256)
cache = BinarySemanticCache(encoder=encoder, max_entries=10000)

# 3. Use
query = "What is the capital of France?"
embedding = backend.embed_text(query)

# Check Cache
if hit := cache.get(embedding):
    print(f"✅ HIT: {hit.response}")
else:
    # Call LLM (Simulated)
    response = "Paris"
    cache.put(embedding, response)
    print(f"❌ MISS: Cached '{response}'")

B. Ollama / Local (Development)

Best for offline development. Zero API costs.

from binary_semantic_cache import BinarySemanticCache, BinaryEncoder
from binary_semantic_cache.embeddings import OllamaEmbedder

# 1. Setup Local Backend (Requires Ollama running with nomic-embed-text)
embedder = OllamaEmbedder(model_name="nomic-embed-text")

# 2. Initialize Cache (768 dimensions for Nomic)
encoder = BinaryEncoder(embedding_dim=768)
cache = BinarySemanticCache(encoder=encoder)

# 3. Use
vec = embedder.embed_text("Hello Local World")
cache.put(vec, "Stored Locally")

📊 Performance

Phase 2.5 introduces a native Rust storage engine, delivering massive gains over the Phase 1 Python baseline.

Latency & Throughput (100k entries)

Memory Efficiency

Note: Actual memory usage depends on the size of your response strings. The cache overhead itself is minimal.

🏗️ Architecture

The cache uses a hybrid Python/Rust architecture to combine ease of use with systems-level performance.

graph LR
    A[User App] -->|Python API| B(BinarySemanticCache)
    B -->|Embed| C{Backend}
    C -->|OpenAI/Ollama| D[Embeddings]
    B -->|Search| E[Rust Core 🦀]
    E -->|SIMD Hamming| F[Binary Index]
    E -->|Results| B

Persistence V3 (Dual-File Format)

Persistence is handled by a split-file strategy ensuring fast loading regardless of cache size:

1. entries.bin: A memory-mappable binary file containing compressed codes, timestamps, and access counts.
   • Index Load Time: < 10ms for 1M entries (search-ready).
   • Full Load Time: ~300ms for 1M entries (includes response hydration).
2. responses.pkl: A standard Python pickle file for storing arbitrary response objects (strings, dicts, JSON).
   • Integrity: Secured with SHA-256 checksums.

⚙️ Configuration

BinarySemanticCache(encoder, max_entries=..., ...)

⚠️ Limitations & Constraints

For a detailed breakdown, see Known Limitations (v1.0).

• Linear Scan (O(N)): This is not an Approximate Nearest Neighbor (ANN) index (like FAISS/HNSW). It performs a full linear scan.
  • Implication: Extremely fast for N < 1M (Rust SIMD), but scales linearly.
• Full Load Time: While the index loads instantly, full hydration of 1M+ response objects takes ~300ms due to Python pickle overhead.
• Memory Resident: The entire index lives in RAM.
  • Implication: 1M entries requires ~50MB RAM + Response Data.
• Global Lock: Uses a global RLock for thread safety.
  • Implication: Concurrent writes are serialized.
• Rust Dependency: You must be able to build Rust extensions to install this library from source (no pre-built wheels yet).

🗺️ Roadmap (Phase 3)

• Cloud Persistence: S3 / GCS adapters for serverless deployments.
• Distributed Cache: Redis-backed shared state for multi-instance setups.
• Approximate Search: Evaluation of HNSW for >1M entry scaling.

🤝 Contributing

We welcome contributions! Please ensure you run the full benchmark suite before submitting PRs.

License: MIT

Maintained by Matteo Panzeri.