isagellm-control-plane 0.4.0.3

sageLLM Control Plane - Intelligent request routing, scheduling, and engine lifecycle management

sageLLM Control Plane

Protocol Compliance (Mandatory)

• MUST follow Protocol v0.1: https://github.com/intellistream/sagellm-docs/blob/main/docs/specs/protocol_v0.1.md
• Any globally shared definitions (fields, error codes, metrics, IDs, schemas) MUST be added to Protocol first.

Intelligent request routing, scheduling, and engine lifecycle management for sageLLM.

Features

• 🎯 Scheduling Policies - FIFO, Priority, SLO-aware, Cost-optimized, Adaptive
• ⚖️ Load Balancing - Intelligent request routing across multiple engine instances
• 📈 Autoscaling - SLA-based autoscaling for Prefill/Decode instances
• 🔄 Engine Lifecycle - Spawn, stop, health check, auto-restart
• 📊 Observability - Metrics collection, performance monitoring
• 🧩 Parallelism - TP, PP, DP, EP strategy optimization

Installation

# Basic installation
pip install isagellm-control-plane

# With optional features
pip install isagellm-control-plane[gpu]      # GPU monitoring
pip install isagellm-control-plane[metrics]  # Prometheus metrics
pip install isagellm-control-plane[all]      # All features

Requirements: Python 3.10+

🚀 开发者快速开始

git clone [e-mail]:intellistream/sagellm-control-plane.git
cd sagellm-control-plane
./quickstart.sh   # 一键安装开发环境（含依赖）

# 或手动安装
pip install -e ".[dev]"

运行测试：

pytest tests/ -v

Quick Start

Running Modes

Mode	Use Case	Backend
CPU	Development/CI	HuggingFace Transformers
GPU	Production	CUDA/Ascend

CPU Mode (Development)

from sagellm_core.llm_engine import LLMEngine, LLMEngineConfig
from sagellm_control import LocalEngineClient
from sagellm_protocol import Request

# Create LLMEngine with TinyLlama (unified hardware-agnostic engine)
config = LLMEngineConfig(
    model_path="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    backend_type="cpu",  # or "cuda", "ascend", "auto"
    max_new_tokens=50,
)
engine = LLMEngine(config)
await engine.start()

# Create local client
client = LocalEngineClient(engine)

# Execute request
request = Request(
    request_id="req-001",
    trace_id="trace-001",
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    prompt="What is AI?",
    max_tokens=30,
)

response = await client.execute_request(request)
print(f"Response: {response.output_text}")
print(f"TTFT: {response.metrics.ttft_ms:.2f}ms")

await engine.stop()

See examples/cpu_engine_demo.py for complete examples.

Execution API

Complete inference execution interface:

from sagellm_control import ControlPlaneManager
from sagellm_protocol import Request

# 使用本地执行器（CPU 模式）
cp = ControlPlaneManager(mode="local")

# 1. 非流式推理
request = Request(
    request_id="req-001",
    trace_id="trace-001",
    model="test-model",
    prompt="Hello, how are you?",
    max_tokens=100,
    stream=False,
)
response = await cp.execute_request(request)
print(f"Output: {response.output_text}")
print(f"TTFT: {response.metrics.ttft_ms:.2f} ms")

# 2. 流式推理
async for event in cp.stream_request(request):
    if event.event == "delta":
        print(event.chunk, end="", flush=True)

# 3. 文本嵌入
embeddings = await cp.get_embeddings(
    texts=["Text 1", "Text 2", "Text 3"],
    model_id="embedding-model"
)
print(f"Generated {len(embeddings)} embeddings of dimension {len(embeddings[0])}")

See examples/execution_layer_demo.py for more examples.

Architecture

sagellm_control/
├── types.py           # Core data types (RequestMetadata, EngineInfo, etc.)
├── strategies/        # Scheduling policies (FIFO, Priority, SLO, etc.)
├── executors/         # Execution coordinators (HTTP, LocalAsync)
├── router.py          # Request routing and load balancing
├── autoscaler.py      # SLA-based autoscaling
├── parallelism.py     # Parallelism strategy optimization
├── manager.py         # Main ControlPlaneManager
└── engine_lifecycle.py # Engine lifecycle management

Documentation

• TODO.md - Development roadmap
• TEAM.md - Team assignments

Related Repositories

• sagellm - Umbrella 包 + CLI
• sagellm-protocol - 协议定义
• sagellm-backend - 后端抽象
• sagellm-gateway - API 网关

---

License

Proprietary - IntelliStream