LlamaIndex LLM Integration for Cloudflare AI Gateway

This package provides integration between LlamaIndex and Cloudflare AI Gateway, allowing you to use multiple AI models from different providers with automatic fallback.

Features

• Multiple Providers: Use OpenAI, Anthropic, Mistral, Groq, and more
• Automatic Fallback: If one provider fails, automatically tries the next
• Streaming Support: Both chat and completion streaming
• Async Support: Full async/await support
• Caching: Built-in request caching with Cloudflare AI Gateway

Installation

pip install llama-index-llms-cloudflare-ai-gateway

Quick Start

from llama_index.llms.cloudflare_ai_gateway import CloudflareAIGateway
from llama_index.llms.openai import OpenAI
from llama_index.llms.anthropic import Anthropic
from llama_index.core.llms import ChatMessage

# Create LLM instances
openai_llm = OpenAI(model="gpt-4o-mini", api_key="your-openai-key")
anthropic_llm = Anthropic(
    model="claude-3-5-sonnet-20241022", api_key="your-anthropic-key"
)

# Create Cloudflare AI Gateway LLM
llm = CloudflareAIGateway(
    llms=[openai_llm, anthropic_llm],  # Try OpenAI first, then Anthropic
    account_id="your-cloudflare-account-id",
    gateway="your-ai-gateway-name",
    api_key="your-cloudflare-api-key",
)

# Use the LLM
messages = [ChatMessage(role="user", content="What is 2+2?")]
response = llm.chat(messages)
print(response.message.content)

Supported Providers

• OpenAI
• Anthropic
• Mistral AI
• Groq
• DeepSeek
• Perplexity
• Replicate
• Grok
• Azure OpenAI

Testing

# Set environment variables
export OPENAI_API_KEY="your-key"
export ANTHROPIC_API_KEY="your-key"
export CLOUDFLARE_ACCOUNT_ID="your-id"
export CLOUDFLARE_API_KEY="your-key"
export CLOUDFLARE_GATEWAY="your-gateway"

# Run tests
uv run pytest tests/