llm-router-utils

Lightweight extraction of sglang's reasoning parser, tool-call parser, and chat template rendering, for use in custom router services and lightweight LLM applications.

This library does not include any inference engine code. It only provides the "frontend" message processing pipeline: OpenAIServingChat._process_messages and its dependencies.

Upstream source: sglang release/v0.5.17.

Installation

pip install llm-router-utils

CPU-only torch (optional, smaller on Linux)

This library depends on xgrammar, which declares torch>=1.10.0. By default pip pulls the CUDA-enabled torch wheel from PyPI (~502 MB on Linux x86_64). Routers don't use torch for GPU ops — only a lazy torch.version.hip / torch.npu.is_available() probe that returns False when torch is absent — so CPU-only torch is sufficient.

To install CPU-only torch, install it from the PyTorch CPU index before this package:

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install llm-router-utils

The second command sees torch already satisfied and skips the CUDA download. On Linux x86_64 this saves ~319 MB (502 → 183). On Windows there is no size difference (both ~116 MB), so this step is only useful on Linux.

Usage

from llm_router_utils.sglang.srt.configs.model_config import ModelConfig
from llm_router_utils.sglang.srt.managers.tokenizer_manager import TokenizerManager
from llm_router_utils.sglang.srt.parser.template_manager import TemplateManager
from llm_router_utils.sglang.srt.entrypoints.openai.serving_chat import OpenAIServingChat
from llm_router_utils.sglang.srt.server_args import ServerArgs, PortArgs

server_args = ServerArgs(model_path="Qwen/Qwen3-32B", tool_call_parser="qwen3_coder")
port_args = PortArgs()
tokenizer_manager = TokenizerManager(server_args, port_args)
template_manager = TemplateManager()
template_manager.initialize_templates(
    tokenizer_manager=tokenizer_manager,
    model_path=server_args.model_path,
    chat_template=server_args.chat_template,
)
openai_serving_chat = OpenAIServingChat(tokenizer_manager, template_manager)
result = openai_serving_chat._process_messages(request, is_multimodal=False)

What's included

Migrated modules under llm_router_utils/sglang/srt/ (paths preserved from upstream):

What's NOT included

Inference engine code is intentionally stripped: schedulers, model loaders, layer implementations, CUDA/Triton kernels, sampling, constrained decoding, speculative decoding, LoRA runtime, distributed runtime, KV cache manager, HTTP server, multimodal processing, observability, and all CLI/launch scripts.

Version Mapping

The mapping between this repo's releases and upstream sglang versions, so users can find the release that matches a given sglang version:

See HOW_TO_UPGRADE.md for the upgrade workflow.

License

Apache 2.0, adapted from sglang.

This is a derivative work of sglang. The upstream copyright notice is retained in LICENSE and NOTICE per Apache 2.0 §4(b)/§4(c). Retained source files keep their upstream copyright headers; slimmed files carry both the upstream header and a "Derivative work" notice. See HOW_TO_UPGRADE.md for the trimming rules.