isagellm-control-plane 0.1.0.4

sageLLM Control Plane - Intelligent request routing, scheduling, and engine lifecycle management

sageLLM Control Plane

Intelligent request routing, scheduling, and engine lifecycle management for sageLLM.

Features

• 🎯 Scheduling Policies - FIFO, Priority, SLO-aware, Cost-optimized, Adaptive
• ⚖️ Load Balancing - Intelligent request routing across multiple engine instances
• 📈 Autoscaling - SLA-based autoscaling for Prefill/Decode instances
• 🔄 Engine Lifecycle - Spawn, stop, health check, auto-restart
• 📊 Observability - Metrics collection, performance monitoring
• 🧩 Parallelism - TP, PP, DP, EP strategy optimization

Installation

# 从 PyPI 安装（基础版）
pip install isagellm-control-plane

# With GPU monitoring
pip install isagellm-control-plane[gpu]

# With Prometheus metrics
pip install isagellm-control-plane[metrics]

# 完整功能（推荐生产环境）
pip install isagellm-control-plane[all]

最低配置要求

Mock 模式（无 GPU 测试）：

• Python 3.10+
• isagellm-protocol (自动安装)
• 内存：< 50 MB
• CPU：任意

CPU 推理模式：

• Python 3.10+
• isagellm-backend 含 CPU 引擎
• 内存：取决于模型（TinyLlama 需 2GB）
• CPU：多核推荐

GPU 推理模式：

• Python 3.10+
• CUDA 11.8+ 或 Ascend CANN 驱动
• 显存：取决于模型（7B 模型需 16GB+）

🚀 开发者快速开始

git clone [e-mail]:intellistream/sagellm-control-plane.git
cd sagellm-control-plane
./quickstart.sh   # 一键安装开发环境（含依赖）

# 或手动安装
pip install -e ".[dev]"

运行测试：

pytest tests/ -v

Quick Start

启动模式说明

Control Plane 支持三种运行模式：

模式	使用场景	依赖	推理能力
Mock	CI/测试/快速验证	无需 GPU	返回固定模拟输出
CPU	本地开发/无 GPU 环境	HuggingFace Transformers	真实推理（较慢）
GPU	生产环境/高性能需求	CUDA/Ascend 驱动	真实推理（高性能）

Fail-Fast 原则：

• 如果配置中指定了后端但无法初始化，Control Plane 会立即抛出错误
• 不会自动降级到 Mock 模式（避免生产环境配置错误被掩盖）
• 符合申报书中的 fail-fast 设计要求

Option 1: Mock Engine (Fast Testing)

from sagellm_control import ControlPlaneManager

# Create manager with mock mode (no GPU required)
manager = ControlPlaneManager(
    scheduling_policy="adaptive",
    routing_strategy="load_balanced",
    mode="local",  # Use local async executor
)

# Register a mock engine
manager.register_engine(
    engine_id="engine-001",
    model_id="mock-model",
    host="localhost",
    port=8000,
)

# Schedule a request
decision = await manager.schedule_request(
    request_id="req-001",
    prompt="Hello, world!",
    max_tokens=128,
)

print(f"Scheduled to: {decision.instance_id}")

Option 2: CPU Engine (Real Inference)

from sagellm_backend.engine.cpu import create_cpu_engine
from sagellm_control import LocalEngineClient
from sagellm_protocol import Request

# Create CPU engine with TinyLlama
engine = create_cpu_engine(
    engine_id="cpu-001",
    model_path="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    max_new_tokens=50,
)
await engine.start()

# Create local client
client = LocalEngineClient(engine)

# Execute request
request = Request(
    request_id="req-001",
    trace_id="trace-001",
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    prompt="What is AI?",
    max_tokens=30,
)

response = await client.execute_request(request)
print(f"Response: {response.output_text}")
print(f"TTFT: {response.metrics.ttft_ms:.2f}ms")

await engine.stop()

See examples/cpu_engine_demo.py for complete examples.

执行层 API (Task0.8)

Control Plane 提供完整的推理执行接口：

from sagellm_control import ControlPlaneManager, MockControlPlane
from sagellm_protocol import Request

# 使用 Mock 模式（无 GPU 依赖）
cp = MockControlPlane()
cp.register_engine("engine-001", model_id="test-model", host="localhost", port=8000)

# 1. 非流式推理
request = Request(
    request_id="req-001",
    trace_id="trace-001",
    model="test-model",
    prompt="Hello, how are you?",
    max_tokens=100,
    stream=False,
)
response = await cp.execute_request(request)
print(f"Output: {response.output_text}")
print(f"TTFT: {response.metrics.ttft_ms:.2f} ms")

# 2. 流式推理
async for event in cp.stream_request(request):
    if event.event == "delta":
        print(event.chunk, end="", flush=True)

# 3. 文本嵌入
embeddings = await cp.get_embeddings(
    texts=["Text 1", "Text 2", "Text 3"],
    model_id="embedding-model"
)
print(f"Generated {len(embeddings)} embeddings of dimension {len(embeddings[0])}")

更多示例请参考 examples/execution_layer_demo.py。

Architecture

sagellm_control/
├── types.py           # Core data types (RequestMetadata, EngineInfo, etc.)
├── strategies/        # Scheduling policies (FIFO, Priority, SLO, etc.)
├── executors/         # Execution coordinators (HTTP, LocalAsync, Mock)
├── router.py          # Request routing and load balancing
├── autoscaler.py      # SLA-based autoscaling
├── parallelism.py     # Parallelism strategy optimization
├── manager.py         # Main ControlPlaneManager
└── engine_lifecycle.py # Engine lifecycle management

Mock-First Development

All modules support mock mode for testing without GPU:

from sagellm_control.executors import MockExecutionCoordinator

# Use mock executor for CI/CD
executor = MockExecutionCoordinator()
result = await executor.execute(request)

---

📚 Documentation

快速链接

• 部署指南 - Control Plane 生产部署
• 故障排查 - 常见问题与解决
• 环境变量 - 配置参考

相关文档

• README.md - 本文档
• TODO.md - 开发任务列表
• TEAM.md - 团队分工
• MOCK_VS_CPU.md - Mock 与 CPU 引擎对比

相关仓库

• sagellm - Umbrella 包 + CLI
• sagellm-protocol - 协议定义
• sagellm-backend - 后端抽象
• sagellm-gateway - API 网关

---

License

Proprietary - IntelliStream