sollol 0.3.6

Super Ollama Load Balancer with Intelligent Routing and Distributed Inference

SOLLOL - Production-Ready Orchestration for Local LLM Clusters

Open-source orchestration layer that combines intelligent task routing with distributed model inference for local LLM clusters.

Quick Start • Features • Architecture • Documentation • Examples

---

🎯 What is SOLLOL?

SOLLOL (Super Ollama Load balancer & Orchestration Layer) transforms your collection of Ollama nodes into an intelligent AI cluster with adaptive routing and automatic failover—all running on your own hardware.

The Problem

You have multiple machines with GPUs running Ollama, but:

• ❌ Manual node selection for each request
• ❌ No way to run models larger than your biggest GPU
• ❌ Can't distribute multi-agent workloads efficiently
• ❌ No automatic failover or load balancing
• ❌ Zero visibility into cluster performance

The SOLLOL Solution

SOLLOL provides:

• ✅ Intelligent routing that learns which nodes work best for each task
• ✅ Model sharding to run 70B+ models across multiple machines
• ✅ Parallel agent execution for multi-agent frameworks
• ✅ Auto-discovery of all nodes and capabilities
• ✅ Built-in observability with real-time metrics
• ✅ Zero-config deployment - just point and go

---

🚀 Why SOLLOL?

1. Two Distribution Modes in One System

SOLLOL combines both task distribution and model sharding:

📊 Task Distribution (Horizontal Scaling)

Distribute multiple requests across your cluster in parallel:

# Run 10 agents simultaneously across 5 nodes
pool = OllamaPool.auto_configure()
responses = await asyncio.gather(*[
    pool.chat(model="llama3.2", messages=[...])
    for _ in range(10)
])
# 5x faster than sequential execution!

🧩 Model Sharding (Vertical Scaling)

Run single large models that don't fit on one machine:

# Run larger models across multiple nodes
# Note: Verified with 13B across 2-3 nodes; larger models not extensively tested
router = HybridRouter(
    enable_distributed=True,
    num_rpc_backends=4
)
response = await router.route_request(
    model="llama3:70b",  # Sharded automatically
    messages=[...]
)

Use them together! Small models use task distribution, large models use sharding.

---

2. Intelligent, Not Just Balanced

SOLLOL doesn't just distribute requests randomly—it learns and optimizes:

Feature	Simple Load Balancer	SOLLOL
Routing	Round-robin	Context-aware scoring
Learning	None	Adapts from performance history
Resource Awareness	None	GPU/CPU/memory-aware
Task Optimization	None	Routes by task type complexity
Failover	Manual	Automatic with health checks
Priority	FIFO	Priority queue with fairness

Example: SOLLOL automatically routes:

• Heavy generation tasks → GPU nodes with 24GB VRAM
• Fast embeddings → CPU nodes or smaller GPUs
• Critical requests → Fastest, most reliable nodes
• Batch processing → Lower priority, distributed load

---

3. Production-Ready from Day One

from sollol import SOLLOL, SOLLOLConfig

# Literally 3 lines to production
config = SOLLOLConfig.auto_discover()
sollol = SOLLOL(config)
sollol.start()  # ✅ Gateway running on :8000

Out of the box:

• Auto-discovery of Ollama nodes
• Health monitoring and failover
• Prometheus metrics
• Web dashboard
• Connection pooling
• Request hedging
• Priority queuing

---

🏗️ Architecture

High-Level Overview

┌────────────────────────────────────────────────────────┐
│                  Your Application                       │
│         (SynapticLlamas, custom agents, etc.)          │
└──────────────────────┬─────────────────────────────────┘
                       │
                       ▼
┌────────────────────────────────────────────────────────┐
│                 SOLLOL Gateway (:8000)                  │
│  ┌──────────────────────────────────────────────────┐  │
│  │         Intelligent Routing Engine               │  │
│  │  • Analyzes: task type, complexity, resources    │  │
│  │  • Scores: all nodes based on context            │  │
│  │  • Learns: from performance history              │  │
│  │  • Routes: to optimal node                       │  │
│  └──────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────┐  │
│  │          Priority Queue + Failover               │  │
│  └──────────────────────────────────────────────────┘  │
└────────┬─────────────────────────┬─────────────────────┘
         │                         │
         ▼                         ▼
  ┌─────────────┐          ┌──────────────┐
  │ Task Mode   │          │  Shard Mode  │
  │ Ray Cluster │          │  llama.cpp   │
  └──────┬──────┘          └──────┬───────┘
         │                         │
         ▼                         ▼
┌────────────────────────────────────────────────────────┐
│              Your Heterogeneous Cluster                 │
│  GPU (24GB) │ GPU (16GB) │ CPU (64c) │ GPU (8GB) │...  │
└────────────────────────────────────────────────────────┘

How Routing Works

# 1. Request arrives
POST /api/chat {
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Complex analysis task..."}],
  "priority": 8
}

# 2. SOLLOL analyzes
task_type = "generation"       # Auto-detected
complexity = "high"             # Token count analysis
requires_gpu = True             # Based on task
estimated_duration = 3.2s       # From history

# 3. SOLLOL scores all nodes
Node A (GPU 24GB, load: 0.2, latency: 120ms) → Score: 185.3 ✓ WINNER
Node B (GPU 8GB,  load: 0.6, latency: 200ms) → Score: 92.1
Node C (CPU only, load: 0.1, latency: 80ms)  → Score: 41.2

# 4. Routes to Node A, monitors execution, learns for next time

Scoring Algorithm:

Score = 100.0 (baseline)
      × success_rate (0.0-1.0)
      ÷ (1 + latency_penalty)
      × gpu_bonus (1.5x if GPU available & needed)
      ÷ (1 + load_penalty)
      × priority_alignment
      × task_specialization

---

📦 Installation

Quick Install (PyPI)

pip install sollol

From Source

git clone https://github.com/BenevolentJoker-JohnL/SOLLOL.git
cd SOLLOL
pip install -e .

---

⚡ Quick Start

1. Synchronous API (No async/await needed!)

New in v0.3.6: SOLLOL now provides a synchronous API for easier integration with non-async applications.

from sollol.sync_wrapper import OllamaPool
from sollol.priority_helpers import Priority

# Auto-discover and connect to all Ollama nodes
pool = OllamaPool.auto_configure()

# Make requests - SOLLOL routes intelligently
# No async/await needed!
response = pool.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello!"}],
    priority=Priority.HIGH,  # Semantic priority levels
    timeout=60  # Request timeout in seconds
)

print(response['message']['content'])
print(f"Routed to: {response.get('_sollol_routing', {}).get('host', 'unknown')}")

Key features of synchronous API:

• ✅ No async/await syntax required
• ✅ Works with synchronous agent frameworks
• ✅ Same intelligent routing and features
• ✅ Runs async code in background thread automatically

---

2. Async API (Original)

For async applications, use the original async API:

from sollol import OllamaPool

# Auto-discover and connect to all Ollama nodes
pool = await OllamaPool.auto_configure()

# Make requests - SOLLOL routes intelligently
response = await pool.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response['message']['content'])
print(f"Routed to: {response['_sollol_routing']['host']}")
print(f"Task type: {response['_sollol_routing']['task_type']}")

---

3. Priority-Based Multi-Agent Execution

New in v0.3.6: Use semantic priority levels and role-based mapping.

from sollol.sync_wrapper import OllamaPool
from sollol.priority_helpers import Priority, get_priority_for_role

pool = OllamaPool.auto_configure()

# Define agents with different priorities
agents = [
    {"name": "Researcher", "role": "researcher"},  # Priority 8
    {"name": "Editor", "role": "editor"},          # Priority 6
    {"name": "Summarizer", "role": "summarizer"},  # Priority 5
]

for agent in agents:
    priority = get_priority_for_role(agent["role"])

    response = pool.chat(
        model="llama3.2",
        messages=[{"role": "user", "content": f"Task for {agent['name']}"}],
        priority=priority
    )
    # User-facing agents get priority, background tasks wait

Priority levels available:

• Priority.CRITICAL (10) - Mission-critical
• Priority.URGENT (9) - Fast response needed
• Priority.HIGH (7) - Important tasks
• Priority.NORMAL (5) - Default
• Priority.LOW (3) - Background tasks
• Priority.BATCH (1) - Can wait

---

4. Model Sharding (Large Models)

from sollol import HybridRouter, OllamaPool

# Enable model sharding for large models
router = HybridRouter(
    ollama_pool=OllamaPool.auto_configure(),
    enable_distributed=True,  # Enable sharding
    auto_discover_rpc=True,   # Find existing RPC backends
    auto_setup_rpc=True,      # Or auto-build them
    num_rpc_backends=3        # Shard across 3 nodes
)

# Use large model that doesn't fit on one machine
response = await router.route_request(
    model="codellama:70b",  # Automatically sharded
    messages=[{"role": "user", "content": "Complex coding task"}]
)

5. SOLLOL Detection

New in v0.3.6: Detect if SOLLOL is running vs native Ollama.

import requests

def is_sollol(url="http://localhost:11434"):
    """Check if SOLLOL is running at the given URL."""

    # Method 1: Check X-Powered-By header
    response = requests.get(url)
    if response.headers.get("X-Powered-By") == "SOLLOL":
        return True

    # Method 2: Check health endpoint
    response = requests.get(f"{url}/api/health")
    data = response.json()
    if data.get("service") == "SOLLOL":
        return True

    return False

# Use it
if is_sollol("http://localhost:11434"):
    print("✓ SOLLOL detected - using intelligent routing")
else:
    print("Native Ollama detected")

Why this matters:

• Enables graceful fallback in client applications
• Makes SOLLOL a true drop-in replacement
• Clients can auto-detect and use SOLLOL features when available

---

6. Production Gateway

from sollol import SOLLOL, SOLLOLConfig

# Full production setup
config = SOLLOLConfig(
    ray_workers=4,
    dask_workers=2,
    hosts=["gpu-1:11434", "gpu-2:11434", "cpu-1:11434"],
    gateway_port=8000,
    metrics_port=9090
)

sollol = SOLLOL(config)
sollol.start()  # Blocks and runs gateway

# Access via HTTP:
# curl http://localhost:8000/api/chat -d '{...}'
# curl http://localhost:8000/api/stats
# curl http://localhost:8000/api/dashboard

---

🎓 Use Cases

1. Multi-Agent AI Systems (SynapticLlamas, CrewAI, AutoGPT)

Problem: Running 10 agents sequentially takes 10x longer than necessary.

Solution: SOLLOL distributes agents across nodes in parallel.

# Before: Sequential execution on one node
# After: Parallel execution with SOLLOL
pool = OllamaPool.auto_configure()
agents = await asyncio.gather(*[
    pool.chat(model="llama3.2", messages=agent_prompts[i])
    for i in range(10)
])
# Speedup depends on number of available nodes and their capacity

2. Large Model Inference

Problem: Your model doesn't fit in available VRAM.

Solution: SOLLOL can shard models across multiple machines via llama.cpp.

# Distribute model across multiple nodes
# Note: Verified with 13B models; larger models not extensively tested
router = HybridRouter(
    enable_distributed=True,
    num_rpc_backends=4
)
# Trade-off: Slower startup/inference but enables running larger models

3. Mixed Workloads

Problem: Different tasks need different resources.

Solution: SOLLOL routes each task to the optimal node.

pool = OllamaPool.auto_configure()

# Heavy generation → GPU node
chat = pool.chat(model="llama3.2:70b", messages=[...])

# Fast embeddings → CPU node
embeddings = pool.embed(model="nomic-embed-text", input=[...])

# SOLLOL automatically routes each to the best available node

4. High Availability Production

Problem: Node failures break your service.

Solution: SOLLOL auto-fails over and recovers.

# Node A fails mid-request
# ✅ SOLLOL automatically:
# 1. Detects failure
# 2. Retries on Node B
# 3. Marks Node A as degraded
# 4. Periodically re-checks Node A
# 5. Restores Node A when healthy

---

📊 Performance & Benchmarks

Task Distribution Performance

Note: These are theoretical estimates based on parallel execution. Actual performance depends on workload characteristics and network latency.

Scenario	Without SOLLOL	With SOLLOL	Estimated Speedup
10 agents, 5 nodes	50s (sequential)	~12s (parallel)	~4x
Mixed workloads	Random routing	Smart routing	20-30% improvement*

*Observed in testing; your results may vary based on cluster configuration.

Model Sharding Performance

Model	Single 24GB GPU	SOLLOL (3×16GB)	Status
13B	✅ ~20 tok/s	✅ ~5 tok/s	✅ Verified working
70B	❌ OOM	⚠️ Estimated ~3-5 tok/s	⚠️ Not extensively tested

When to use sharding: When model doesn't fit on your largest GPU. You trade speed for capability.

Performance trade-offs: Distributed inference is 2-5 minutes slower to start and ~4x slower for inference compared to local. Use only when necessary.

Overhead

• Routing decision: ~5-10ms (tested with 5-10 nodes)
• Network overhead: Varies by network (typically 5-20ms)
• Total added latency: ~20-50ms
• Benefit: Better resource utilization + automatic failover

---

🛠️ Advanced Configuration

Custom Routing Strategy

from sollol import OllamaPool

pool = OllamaPool(
    nodes=[
        {"host": "gpu-1.local", "port": 11434, "priority": 10},  # Prefer this
        {"host": "gpu-2.local", "port": 11434, "priority": 5},
        {"host": "cpu-1.local", "port": 11434, "priority": 1},   # Last resort
    ],
    enable_intelligent_routing=True,
    enable_hedging=True,  # Duplicate critical requests
    max_queue_size=100
)

Priority-Based Scheduling

# Critical user-facing request
response = pool.chat(
    model="llama3.2",
    messages=[...],
    priority=10  # Highest priority
)

# Background batch job
response = pool.chat(
    model="llama3.2",
    messages=[...],
    priority=1  # Lowest priority
)

# SOLLOL ensures high-priority requests jump the queue

Observability & Monitoring

# Get detailed stats
stats = pool.get_stats()
print(f"Total requests: {stats['total_requests']}")
print(f"Average latency: {stats['avg_latency_ms']}ms")
print(f"Success rate: {stats['success_rate']:.2%}")

# Per-node breakdown
for host, metrics in stats['hosts'].items():
    print(f"{host}: {metrics['latency_ms']}ms, {metrics['success_rate']:.2%}")

# Prometheus metrics endpoint
curl http://localhost:9090/metrics

# sollol_requests_total{host="gpu-1:11434",model="llama3.2"} 1234
# sollol_latency_seconds{host="gpu-1:11434"} 0.234
# sollol_success_rate{host="gpu-1:11434"} 0.98

---

🔌 Integration Examples

SynapticLlamas Integration

from sollol import SOLLOL, SOLLOLConfig
from synaptic_llamas import AgentOrchestrator

# Setup SOLLOL for multi-agent orchestration
config = SOLLOLConfig.auto_discover()
sollol = SOLLOL(config)
sollol.start(blocking=False)

# SynapticLlamas now uses SOLLOL for intelligent routing
orchestrator = AgentOrchestrator(
    llm_endpoint="http://localhost:8000/api/chat"
)

# All agents automatically distributed and optimized
orchestrator.run_parallel_agents([...])

LangChain Integration

from langchain.llms import Ollama
from sollol import OllamaPool

# Use SOLLOL as LangChain backend
pool = OllamaPool.auto_configure()

llm = Ollama(
    base_url="http://localhost:8000",
    model="llama3.2"
)

# LangChain requests now go through SOLLOL
response = llm("What is quantum computing?")

---

📚 Documentation

• Architecture Guide - Deep dive into system design
• Integration Examples - Practical integration patterns
  • Synchronous Agent Integration
  • Priority Configuration
  • Load Balancer Wrapper
• Deployment Guide - Production deployment patterns
• API Reference - Complete API documentation
• Performance Tuning - Optimization guide
• SynapticLlamas Learnings - Features from production use

---

🆕 What's New in v0.3.6

Synchronous API

No more async/await required! SOLLOL now provides a synchronous API wrapper that works with traditional Python applications and agent frameworks.

from sollol.sync_wrapper import OllamaPool, HybridRouter

pool = OllamaPool.auto_configure()  # No await needed
response = pool.chat(...)            # Synchronous call

Priority Helpers

Semantic priority levels and role-based mapping make priority configuration much easier:

from sollol.priority_helpers import Priority, get_priority_for_role

# Use semantic constants
priority = Priority.HIGH  # 7

# Or map from agent roles
priority = get_priority_for_role("researcher")  # 8

SOLLOL Detection

Clients can now detect if SOLLOL is running vs native Ollama via:

• X-Powered-By: SOLLOL header on all responses
• /api/health endpoint returns {"service": "SOLLOL", "version": "0.3.6"}

Integration Examples

Comprehensive examples showing:

• Synchronous agent integration patterns
• Priority configuration and mapping
• Wrapping SOLLOL around existing infrastructure
• Gradual migration from legacy systems

---

🆚 Comparison

SOLLOL vs. Simple Load Balancers

Feature	nginx/HAProxy	SOLLOL
Routing	Round-robin/random	Context-aware, adapts from history
Resource awareness	None	GPU/CPU/memory-aware
Failover	Manual config	Automatic detection & recovery
Model sharding	❌	✅ llama.cpp integration
Task prioritization	❌	✅ Priority queue
Observability	Basic	Rich metrics + dashboard
Setup	Complex config	Auto-discover

SOLLOL vs. Kubernetes

Feature	Kubernetes	SOLLOL
Complexity	High - requires cluster setup	Low - pip install
AI-specific	Generic container orchestration	Purpose-built for LLMs
Intelligence	None	Task-aware routing
Model sharding	Manual	Automatic
Best for	Large-scale production	AI-focused teams

Use both! Deploy SOLLOL on Kubernetes for ultimate scalability.

---

🤝 Contributing

We welcome contributions! Areas we'd love help with:

• ML-based routing predictions
• Additional monitoring integrations
• Cloud provider integrations
• Performance optimizations
• Documentation improvements

See CONTRIBUTING.md for guidelines.

---

📜 License

MIT License - see LICENSE file for details.

---

🙏 Credits

Created by BenevolentJoker-JohnL

Part of the SynapticLlamas ecosystem.

Built with: Ray, Dask, FastAPI, llama.cpp, Ollama

---

🎯 What Makes SOLLOL Different?

1. Combines task distribution AND model sharding in one system
2. Context-aware routing that adapts based on performance metrics
3. Auto-discovery of nodes with minimal configuration
4. Built-in failover and priority queuing
5. Purpose-built for Ollama clusters (understands GPU requirements, task types)

Limitations to know:

• Model sharding verified with 13B models; larger models not extensively tested
• Performance benefits depend on network latency and workload patterns
• Not a drop-in replacement for single-node setups in all scenarios

---

Stop manually managing your LLM cluster. Let SOLLOL optimize it for you.

Get Started • View on GitHub • Report Issue