multiplexer-llm 0.2.3

A multiplexer for Large Language Model APIs built on the OpenAI SDK. It combines quotas from multiple models and automatically uses fallback models when the primary models are rate limited.

Multiplexer LLM (Python)

Unlock the Power of Distributed AI 🚀

A lightweight Python library that combines the quotas of multiple open source LLM providers with a single unified API. Seamlessly distribute your requests across various providers hosting open source models, ensuring maximum throughput and reliability.

The Problem: Limited AI Resources

• ❌ Rate Limit Errors: "Rate limit exceeded" errors hinder your application's performance
• ❌ Limited Throughput: Single provider constraints limit your AI capabilities
• ❌ Unpredictable Failures: Rate limits can occur at critical moments
• ❌ Manual Intervention: Switching providers requires code changes

The Solution: Unified Access to Multiple Providers

• ✅ Increased Throughput: Combine quotas from multiple open source LLM providers
• ✅ Error Resilience: Automatic failover when one provider hits rate limits
• ✅ Seamless Integration: Compatible with OpenAI SDK for easy adoption
• ✅ Smart Load Balancing: Weight-based distribution across providers for optimal performance

Key Benefits

• 🚀 Scalable AI: Combine resources from multiple providers for enhanced capabilities
• 🛡️ Error Prevention: Automatic failover minimizes rate limit failures
• ⚡ High Availability: Seamless switching between providers ensures continuous operation
• 🔌 OpenAI SDK Compatibility: Works with existing OpenAI SDK code
• 📊 Usage Analytics: Track provider performance and rate limits

How It Works

Single Model:        [Model A: 10K RPM] ❌ Rate Limit Error at 10,001 requests
Multiple Providers:  [Provider 1: 10K] + [Provider 2: 15K] + [Provider 3: 20K] = 45,000 RPM ✅
Multiple Models:     [Model A: 10K] + [Model B: 50K] + [Model C: 15K] = 75,000 RPM ✅✅

Installation

pip install multiplexer-llm

The package requires Python 3.8+ and automatically installs the OpenAI Python SDK as a dependency.

Quick Start

import asyncio
import os
from multiplexer_llm import Multiplexer
from openai import AsyncOpenAI

async def main():
    # Create client instances for a few open source models
    model1 = AsyncOpenAI(
        api_key=os.getenv("MODEL1_API_KEY"),
        base_url="https://api.model1.com/v1/",
    )

    model2 = AsyncOpenAI(
        api_key=os.getenv("MODEL2_API_KEY"),
        base_url="https://api.model2.org/v1",
    )

    # Initialize multiplexer
    async with Multiplexer() as multiplexer:
        # Add models with weights
        multiplexer.add_model(model1, 5, "model1-large")
        multiplexer.add_model(model2, 3, "model2-base")

        # Use like a regular OpenAI client
        completion = await multiplexer.chat.completions.create(
            model="placeholder",  # Will be overridden by selected model
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": "What is the capital of France?"},
            ],
        )

        print(completion.choices[0].message.content)
        print("Model usage stats:", multiplexer.get_stats())

# Run the async function
asyncio.run(main())

How Primary and Fallback Models Work

The multiplexer operates with a two-tier system:

Primary Models (add_model)

• First choice: Used when available
• Weight-based selection: Higher weights = higher probability of selection

Fallback Models (add_fallback_model)

• Backup safety net: Activated when all primary models hit rate limits

API Examples

Creating a Multiplexer

from multiplexer_llm import Multiplexer

# Create multiplexer instance
multiplexer = Multiplexer()

# Or use as async context manager (recommended)
async with Multiplexer() as multiplexer:
    # Your code here
    pass

Adding Models

# Add a primary model
multiplexer.add_model(client: AsyncOpenAI, weight: int, model_name: str)

# Add a fallback model
multiplexer.add_fallback_model(client: AsyncOpenAI, weight: int, model_name: str)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

About Haven Network

Haven Network builds open-source tools to help online communities produce high-quality data for multi-modal AI, with a strong focus on local inference and data privacy.