sageLLM
Protocol Compliance (Mandatory)
- MUST follow Protocol v0.1: https://github.com/intellistream/sagellm-docs/blob/main/docs/specs/protocol_v0.1.md
- Any globally shared definitions (fields, error codes, metrics, IDs, schemas) MUST be added to Protocol first.
🚀 Modular LLM Inference Engine for Domestic Computing Power
Ollama-like experience for Chinese hardware ecosystems (Huawei Ascend, NVIDIA)
✨ Features
- 🎯 One-Click Install -
pip install isagellmgets you started immediately - 🧠 CPU-First - Default CPU engine, no GPU required
- 🇨🇳 Domestic Hardware - First-class support for Huawei Ascend NPU
- 📊 Observable - Built-in metrics (TTFT, TBT, throughput, KV usage)
- 🧩 Plugin System - Extend with custom backends and engines
- 🔄 Mixed Inference - Unified LLM + Embedding client (
MixedInferenceClient) - 🦙 Ollama Backend - Use a local Ollama server as an inference backend
- 📈 Performance Profiling - Load profiling data and interpolate TTFT/throughput
性能主路径规范
国产硬件性能优化的唯一主路径规范已收敛到 sagellm-docs：
- canonical spec: https://github.com/intellistream/sagellm-docs/blob/main/docs/specs/performance_mainline_architecture.md
- 执行语义必须落在
scheduler -> executor -> worker -> model_runner -> backend attention/kernel主链上。 shared stream只负责 admission、telemetry、event fan-out，不能长期维护第二套 decode 语义。vLLM与 HuggingFace fallback 只用于 benchmark、correctness 或 compatibility，不是 SageLLM 的长期性能方向。
📦 Quick Install
# Install sageLLM (recommended, includes gateway/control-plane/kv/comm/compression)
pip install isagellm
# 安装后推荐先运行交互式向导（自动检测机器并推荐模型）
sagellm-setup
# 或
sagellm setup
# Optional: add embedding toolkit (sentence-transformers)
pip install 'isagellm[full]'
# Reproduce exactly-tested sub-package versions (recommended for production)
pip install isagellm -c https://raw.githubusercontent.com/intellistream/sagellm/main-dev/constraints.txt
🚀 安装 PyTorch（推荐）
sagellm install <backend> 会优先走官方兼容矩阵；只有当对应 GitHub wheel 已发布并验证过时，--github 才会启用加速源。
# 方法 1：使用 sagellm CLI (推荐，最简单)
pip install isagellm
sagellm install cuda # 官方后端安装/修复入口
sagellm install cuda --github # 若存在已验证的 GitHub wheel，则自动走加速源
sagellm install cuda # 官方 PyTorch 源（默认）
# 方法 2：直接使用官方 PyTorch index
pip install torch torchvision torchaudio \
--index-url https://download.pytorch.org/whl/cu128
其他支持的后端：
sagellm install ascend- 华为昇腾 NPUsagellm install kunlun- 百度昆仑 XPUsagellm install haiguang- 海光 DCUsagellm install cpu- CPU-only（最小下载）
💡 为什么使用 GitHub 加速？
- ✅ 国内访问速度快（GitHub CDN）
- ✅ 无需配置镜像源
- ✅ 官方 wheels，100% 可信
📦 Wheels 仓库: https://github.com/intellistream/sagellm-pytorch-wheels
🚀 Quick Start
开发者环境变量（private 仓库必需）
quickstart.sh 在 dev 模式下需要 clone sagellm-* 私有仓库，请先配置 GITHUB_TOKEN：
cd sagellm
cp .env.template .env
# 编辑 .env，填入 GITHUB_TOKEN
./quickstart.sh
如果网络环境下 HTTPS clone 不稳定（如 HTTP/2 中断），可切换 SSH：
./quickstart.sh --clone-protocol ssh
若仅需安装 umbrella 包且不 clone 子仓库，可使用：
./quickstart.sh --standard
CLI 命令统一
- 统一主命令：
sagellm
CLI (像 vLLM/Ollama 一样简单)
# 一键启动（完整栈：Gateway + Engine）
pip install isagellm
sagellm serve --model Qwen2-7B
# ✅ OpenAI API 自动可用
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2-7B",
"messages": [{"role": "user", "content": "Hello!"}]
}'
# 查看系统信息
sagellm info
# 单次推理（不启动服务器）
sagellm run -p "What is LLM inference?"
# 默认是安静模式（动态“正在思考”光标）；如需详细日志用 --debug
sagellm run -p "What is LLM inference?" --debug
sagellm chat --debug
# 推荐生产启动（通过 Gateway + Control Plane）
sagellm serve --backend cpu --model sshleifer/tiny-gpt2 --port 8888
sagellm serve \
--backend cpu \
--model sshleifer/tiny-gpt2 \
--port 8888 \
--with-embedding \
--embedding-model sentence-transformers/all-MiniLM-L6-v2
# 一键启动：Gateway + LLM + Embedding
sagellm serve \
--backend cpu \
--model sshleifer/tiny-gpt2 \
--port 8888 \
--with-embedding \
--embedding-model sentence-transformers/all-MiniLM-L6-v2
# 生产推荐：上线前模型健康防护（预检 + 启动检查 + 周期巡检 + 备用模型）
export SAGELLM_PREFLIGHT_CANARY=1
export SAGELLM_STARTUP_CANARY=1
export SAGELLM_PERIODIC_CANARY=1
export SAGELLM_FALLBACK_MODEL="Qwen/Qwen2.5-0.5B-Instruct"
export SAGELLM_CANARY_INTERVAL_SEC=300
export SAGELLM_CANARY_FAIL_THRESHOLD=3
sagellm serve \
--backend pytorch-cuda \
--model Qwen/Qwen2.5-1.5B-Instruct \
--port 8000
🛡️ 生产健康防护（Canary + Fallback）
sagellm serve 支持上线安全防护，避免“模型已损坏但服务仍对外提供垃圾输出”：
- Preflight Canary（默认开启）：服务启动前，本地加载并测试主模型；失败则自动尝试备用模型。
- Startup Canary（默认开启）：Engine 健康后，立即发送固定测试请求；失败则 fail-fast 退出。
- Periodic Canary（默认开启）：后台定期巡检输出质量；连续失败达到阈值后熔断退出（交由 supervisor 重启）。
可用环境变量：
SAGELLM_PREFLIGHT_CANARY：是否启用启动前预检（默认1）SAGELLM_STARTUP_CANARY：是否启用启动后一次性检查（默认1）SAGELLM_PERIODIC_CANARY：是否启用周期巡检（默认1）SAGELLM_FALLBACK_MODEL：备用模型列表（逗号分隔，按顺序回退）SAGELLM_CANARY_INTERVAL_SEC：周期巡检间隔秒数（默认300）SAGELLM_CANARY_FAIL_THRESHOLD：连续失败熔断阈值（默认3）
示例（主模型异常时自动切换到 0.5B）：
export SAGELLM_FALLBACK_MODEL="Qwen/Qwen2.5-0.5B-Instruct"
sagellm serve --model Qwen/Qwen2.5-1.5B-Instruct
Python API (Control Plane - Recommended)
import asyncio
from sagellm import ControlPlaneManager, BackendConfig, EngineConfig
# Install with: pip install isagellm
async def main() -> None:
manager = ControlPlaneManager(
backend_config=BackendConfig(kind="cpu", device="cpu"),
engine_configs=[
EngineConfig(
kind="cpu",
model="sshleifer/tiny-gpt2",
model_path="sshleifer/tiny-gpt2"
)
]
)
await manager.start()
try:
# Requests are automatically routed to available engines
response = await manager.execute_request(
prompt="Hello, world!",
max_tokens=128
)
print(response.output_text)
print(f"TTFT: {response.metrics.ttft_ms:.2f} ms")
print(f"Throughput: {response.metrics.throughput_tps:.2f} tokens/s")
finally:
await manager.stop()
asyncio.run(main())
⚠️ Important: Direct engine creation (create_engine()) is not exported from the umbrella
package. All production code must use ControlPlaneManager for proper request routing, scheduling,
and lifecycle management.
Mixed Inference (LLM + Embedding)
from sagellm import MixedInferenceClient, MixedRequest, RequestKind
# Unified client for both LLM and embedding
client = MixedInferenceClient(
llm_url="http://localhost:8000",
embedding_url="http://localhost:8001",
)
# LLM completion
resp = client.complete("What is 2+2?")
print(resp["text"])
# Embedding
vecs = client.embed("Hello world")
# Mixed batch dispatch
results = client.dispatch([
MixedRequest(kind=RequestKind.LLM, content="Tell me a joke"),
MixedRequest(kind=RequestKind.EMBEDDING, content="The quick brown fox"),
])
Ollama Backend
# Use a local Ollama server as inference backend
sagellm ollama status # check health
sagellm ollama list # list models
sagellm ollama run -m llama3 -p "Hello!" # single completion
sagellm ollama chat -p "Explain Python" # chat
from sagellm import OllamaClient
client = OllamaClient(model="llama3")
resp = client.complete("What is 2+2?")
print(resp["text"])
models = client.list_models()
Performance Profiling & Interpolation
from sagellm.profiling import PerformanceInterpolator
# Load CSV: columns isl, ttft, itl, throughput
interp = PerformanceInterpolator.from_csv("profiles/qwen2_7b_a100.csv")
# Predict metrics for a given input sequence length
ttft = interp.predict_ttft(512) # → seconds
itl = interp.predict_itl(512) # → seconds/token
tput = interp.predict_throughput(512) # → tokens/second
# Reverse: find max ISL that satisfies a TTFT budget
max_isl = interp.reverse_ttft(target_ttft=0.3)
print(f"Max ISL for 300ms TTFT: {max_isl} tokens")
Configuration
# ~/.sagellm/config.yaml
backend:
kind: cpu # Options: cpu, pytorch-cuda, pytorch-ascend
device: cpu
engine:
kind: cpu
model: sshleifer/tiny-gpt2
control_plane:
endpoint: "localhost:8080"
📊 Metrics & Validation
sageLLM provides comprehensive performance metrics:
{
"ttft_ms": 45.2,
"tbt_ms": 12.5,
"throughput_tps": 80.0,
"peak_mem_mb": 24576,
"kv_used_tokens": 4096,
"prefix_hit_rate": 0.85
}
Run benchmarks:
sagellm demo --workload year1 --output metrics.json
🏗️ Architecture
isagellm (umbrella package)
├── isagellm-protocol # Protocol v0.1 types
│ └── Request, Response, Metrics, Error, StreamEvent
├── isagellm-backend # Hardware abstraction (L1 - Foundation)
│ └── BackendProvider, CPUBackend, (CUDABackend, AscendBackend)
├── isagellm-comm # Communication primitives (L2 - Infrastructure)
│ └── Topology, CollectiveOps (all_reduce/gather), P2P (send/recv), Overlap
├── isagellm-kv-cache # KV cache management (L2 - Optional)
│ └── PrefixCache, MemoryPool, EvictionPolicies, Predictor, KV Transfer
├── isagellm-compression # Inference acceleration (quantization, sparsity, etc.) (L2 - Optional)
│ └── Quantization, Sparsity, SpeculativeDecoding, Fusion
├── isagellm-core # Engine core & runtime (L3)
│ └── Config, Engine, Factory, DemoRunner, Native runtime, Plugin abstractions
├── isagellm-control-plane # Request routing & scheduling (L4 - Optional)
│ └── ControlPlaneManager, Router, Policies, Lifecycle
└── isagellm-gateway # OpenAI-compatible REST API (L5 - Optional)
└── FastAPI server, /v1/chat/completions, Session management
🔧 Development
Quick Setup (Development Mode)
# Clone all repositories
./scripts/clone-all-repos.sh
# 默认 dev 模式：安装 sagellm + 子仓库（editable）
./quickstart.sh
# standard 模式：子仓库走 PyPI，sagellm 本体保持本地 editable
./quickstart.sh --standard
# 非交互模式（CI/脚本）
./quickstart.sh --yes
# 可选：跳过清理旧 isagellm* 包
./quickstart.sh --skip-cleanup
# 可选：镜像控制
./quickstart.sh --use-mirror auto
./quickstart.sh --no-mirror
# Open all repos in VS Code Multi-root Workspace
code sagellm.code-workspace
环境说明：
- 不允许在
venv/.venv中运行 quickstart。 - 推荐使用已有 Conda 环境。
- 若使用系统 Python，脚本会进行确认（
--yes可自动确认）。
📖 See WORKSPACE_GUIDE.md for Multi-root Workspace usage.
Testing
# Clone and setup
git clone https://github.com/IntelliStream/sagellm.git
cd sagellm
pip install -e ".[dev]"
### ⚠️ GitHub Actions 账单阻塞时的本地替代 CI
如果 Actions 因 billing/quota 无法启动，可在仓库根目录执行：
```bash
bash scripts/local_ci_fallback.sh
脚本会按 ci.yml 的核心顺序执行（pre-commit、version-check、CPU 单测/集成测试、CLI smoke、build+twine check），用于 issue/PR
附可复现结论。
Run tests
pytest -v
Format & lint
ruff format . ruff check . --fix
Type check
mypy src/sagellm/
Verify dependency hierarchy
python scripts/verify_dependencies.py
### 📖 Development Resources
- **[DEPLOYMENT_GUIDE.md](docs/DEPLOYMENT_GUIDE.md)** - 完整部署与配置指南
- **[TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md)** - 故障排查快速参考
- **[ENVIRONMENT_VARIABLES.md](docs/ENVIRONMENT_VARIABLES.md)** - 环境变量完整参考
- **[DEVELOPER_GUIDE.md](docs/DEVELOPER_GUIDE.md)** - 开发者指南
- **[WORKSPACE_GUIDE.md](docs/WORKSPACE_GUIDE.md)** - Multi-root Workspace 使用
- **[INFERENCE_FLOW.md](docs/INFERENCE_FLOW.md)** - 推理流程详解
- **[PR_CHECKLIST.md](docs/PR_CHECKLIST.md)** - Pull Request 检查清单
______________________________________________________________________
## 📚 Documentation Index
### 用户文档
- [快速开始](README.md#-quick-start) - 5 分钟上手
- [部署指南](docs/DEPLOYMENT_GUIDE.md) - 生产环境部署
- [配置参考](docs/DEPLOYMENT_GUIDE.md#%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6%E8%AF%B4%E6%98%8E) - 完整配置选项
- [环境变量](docs/ENVIRONMENT_VARIABLES.md) - 环境变量参考
- [故障排查](docs/TROUBLESHOOTING.md) - 常见问题解决
### 开发者文档
- [开发指南](docs/DEVELOPER_GUIDE.md) - 贡献代码
- [架构设计](README.md#-architecture) - 系统架构
- [Workspace 使用](docs/WORKSPACE_GUIDE.md) - Multi-root 工作区
- [PR 检查清单](docs/PR_CHECKLIST.md) - 提交前检查
### API 文档
- OpenAI 兼容 API - 参见 [sagellm-gateway](https://github.com/intellistream/sagellm-gateway)
- Python API - 参见 [API_REFERENCE.md](docs/API_REFERENCE.md)（待补充）
### 子包文档
- [sagellm-protocol](https://github.com/intellistream/sagellm-protocol) - 协议定义
- [sagellm-backend](https://github.com/intellistream/sagellm-backend) - 后端抽象
- [sagellm-core](https://github.com/intellistream/sagellm-core) - 引擎核心
- [sagellm-control-plane](https://github.com/intellistream/sagellm-control-plane) - 控制面
- [sagellm-gateway](https://github.com/intellistream/sagellm-gateway) - API 网关
- [sagellm-benchmark](https://github.com/intellistream/sagellm-benchmark) - 基准测试
- [**DEVELOPER_GUIDE.md**](DEVELOPER_GUIDE.md) - 架构规范与开发指南
- [**PR_CHECKLIST.md**](PR_CHECKLIST.md) - Pull Request 审查清单
- [**scripts/verify_dependencies.py**](scripts/verify_dependencies.py) - 依赖层次验证
## � 贡献指南
### 工作流程（必须遵循）
在提交代码前，**必须**严格遵循以下步骤：
#### 1️⃣ 创建 Issue
描述你要解决的问题、实现的功能或改进：
```bash
gh issue create \
--title "[Category] 简短描述" \
--label "bug,sagellm-core" \
--body "详细描述..."
Issue 类型：
[Bug]- Bug 修复[Feature]- 新功能[Performance]- 性能优化[Integration]- 与其他模块集成[Docs]- 文档改进
2️⃣ 在本地分支开发
创建开发分支并解决问题：
# 从 main-dev 创建分支（不是 main！）
git fetch origin main-dev
git checkout -b fix/#123-short-description origin/main-dev
# 进行开发
# ...
# 确保通过所有检查
ruff format .
ruff check . --fix
pytest -v
分支命名约定：
- Bug 修复：
bugfix/#123-xxx - 新功能：
feature/#456-xxx - 文档：
docs/#789-xxx - 性能：
perf/#101-xxx
3️⃣ 发起 Pull Request
提交代码供审查：
git push origin fix/#123-short-description
gh pr create \
--base main-dev \
--head fix/#123-short-description \
--title "Fix: [简短描述]" \
--body "解决 #123
## 改动
- 改动 1
- 改动 2
## 测试
- 新增单元测试
- 所有测试通过 ✓"
PR 必须包含：
- 清晰的标题（Fix/Feature/Docs/Perf）
- 关联 issue 号：
Closes #123 - 改动列表和测试说明
- 通过所有 CI 检查
4️⃣ 代码审查与合并
等待审批后合并到 main-dev：
# 在 GitHub 界面点击"Merge"按钮
# 合并到 main-dev（不是 main！）
合并前条件：
- ✅ 至少一名维护者审批
- ✅ CI 检查全部通过（pytest, ruff）
- ✅ 合并到
main-dev分支
快速检查清单
在发起 PR 前检查：
- 从
main-dev分支创建开发分支 - 更新了
CHANGELOG.md -
ruff format .格式化代码 -
ruff check . --fix通过 lint -
pytest -v通过所有测试 - 关联了相关 issue：
Closes #123
反面例子 ❌
- ❌ 直接在
main分支提交 - ❌ PR 中没有关联 issue
- ❌ 修改了代码但没有更新 CHANGELOG
- ❌ 代码没有通过 lint 检查
- ❌ 提交前没有运行测试
相关资源
- Issue Labels：
bug,enhancement,documentation,sagellm-core,sagellm-backend等 - GitHub CLI：
gh issue create,gh pr create - 更多信息：见
.github/copilot-instructions.md
�📚 Package Details
| Package | PyPI Name | Import Name | Description |
|---|---|---|---|
| sagellm | isagellm |
sagellm |
Umbrella package (install this) |
| sagellm-protocol | isagellm-protocol |
sagellm_protocol |
Protocol v0.1 types |
| sagellm-core | isagellm-core |
sagellm_core |
Runtime & config |
| sagellm-backend | isagellm-backend |
sagellm_backend |
Hardware abstraction |
📄 License
Proprietary - IntelliStream. Internal use only.
Built with ❤️ by IntelliStream Team for domestic AI infrastructure
# test1 maintainer
Metadata
1 maintainer
Release files for isagellm 0.5.4.70
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| isagellm-0.5.4.70.tar.gz | 127.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| isagellm-0.5.4.70-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size: 249.5 kB
Release files / isagellm-0.5.4.70.tar.gz
| Download URL | isagellm-0.5.4.70.tar.gz |
|---|---|
| Size | 127.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a376275d8a30ac37e35d2c3cc2e5ecdfd8ed2d85b6d571042b43fafdeec77e6e
|
|
BLAKE2b-256 checksum How to use checksums |
003de0d93cd2445bdac16d80029e0f9db7c6615b16457449d24a204bcf2f5460
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.15
|
Release files / isagellm-0.5.4.70-py2.py3-none-any.whl
| Download URL | isagellm-0.5.4.70-py2.py3-none-any.whl |
|---|---|
| Size | 121.8 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
6714f1587770e16014006a76c4a0672ff2e9bf93312218a783af927823b754b3
|
|
BLAKE2b-256 checksum How to use checksums |
102b6412cf12182142037e1ef7bab0f4a9e3ab018de8ef915d98d9953928825c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.15
|
