bento-sgl-router 0.0.6

Bento service with SGLang Router included

bento-sgl-router

Helper utilities for bundling an SGLang Router inside a BentoML service. It exposes a drop-in replacement for bentoml.service that starts the router sidecar, keeps it in sync with your LLM workers, and surfaces router metrics alongside your Bento.

Overview

• Wraps an existing BentoML service definition and injects an sglang-router process.
• Automatically syncs worker hosts discovered from your llm dependency during readiness probes.
• Exposes router Prometheus metrics by merging them into the Bento service metrics endpoint.

Installation

pip install bento-sgl-router

Python 3.10 or newer is required. The package installs bentoml>=1.4.25 and sglang-router>=0.1.9 as dependencies.

Quickstart

Decorate an existing BentoML LLM service to bundle it with an SGLang Router:

import bentoml
from bento_sgl_router import service


@service(name="chat-router")
class ChatLLM:
    ...

Any keyword arguments supported by bentoml.service (such as image, labels, or resource configs) can be passed through the decorator.

Router configuration

Control router behaviour via environment variables:

Variable	Default	Description
ROUTER_POLICY	cache_aware	Scheduling policy passed to sglang_router.launch_router.

Set the variable in your deployment environment or via envs=[...] when declaring the service.

Development

pdm install
pdm run pytest

The project keeps the code in bento_sgl_router.py and lightweight tests under tests/.