llm-router-utils

Lightweight extraction of sglang's reasoning parser, tool-call parser, and chat template rendering, for use in custom router services and lightweight LLM applications.

This library does not include any inference engine code. It only provides the "frontend" message processing pipeline: OpenAIServingChat._process_messages and its dependencies.

Upstream source: sglang release/v0.5.16.

Installation

pip install llm-router-utils

Usage

from llm_router_utils.sglang.srt.configs.model_config import ModelConfig
from llm_router_utils.sglang.srt.managers.tokenizer_manager import TokenizerManager
from llm_router_utils.sglang.srt.parser.template_manager import TemplateManager
from llm_router_utils.sglang.srt.entrypoints.openai.serving_chat import OpenAIServingChat
from llm_router_utils.sglang.srt.server_args import ServerArgs, PortArgs

server_args = ServerArgs(model_path="Qwen/Qwen3-32B", tool_call_parser="qwen3_coder")
port_args = PortArgs()
tokenizer_manager = TokenizerManager(server_args, port_args)
template_manager = TemplateManager()
template_manager.initialize_templates(
    tokenizer_manager=tokenizer_manager,
    model_path=server_args.model_path,
    chat_template=server_args.chat_template,
)
openai_serving_chat = OpenAIServingChat(tokenizer_manager, template_manager)
result = openai_serving_chat._process_messages(request, is_multimodal=False)

What's included

Migrated modules under llm_router_utils/sglang/srt/ (paths preserved from upstream):

Module Description
parser/ Conversation templates (~50 model families), ReasoningParser + detectors, harmony/inkling, jinja utils, template detection, TemplateManager with TokenizerLike Protocol
function_call/ FunctionCallParser + 33 detectors (hermes, glm, deepseek, qwen, kimi, mistral, …), JsonArrayParser, schema utils
entrypoints/openai/ protocol.py (~1900 lines, full OpenAI types), slimmed serving_chat.py (only _process_messages chain), serving_base.py, chat_encoding.py, encoding_dsv32/dsv4.py, sse_utils.py, usage_processor.py, utils.py
managers/ Slimmed TokenizerManager (uses upstream get_tokenizer for byte-parity incl. SGLANG_PATCH_TOKENIZER), slimmed io_struct.py, embed_types.py stub
configs/ Slimmed ModelConfig (uses upstream get_config; exposes hf_config/is_multimodal/get_default_sampling_params/context_length), model_config_parser_registry.py
tokenizer/ tiktoken_tokenizer.py
disaggregation/ kv_events.py — KV cache event structs (EventBatch, KVCacheEvent, StorageMedium, BlockStored, BlockRemoved, AllBlocksCleared, KVEventBatch)
mem_cache/ utils.py — pure-Python SHA256 hash helpers, byte-identical to sglang's C++ extension
observability/ metrics_collector.py — data classes only: QueueCount, SchedulerStats, compute_routing_key_stats. Heavy *MetricsCollector classes stripped
utils/hf_transformers/ Restored common.py/config.py/tokenizer.py/mistral_utils.py — upstream get_tokenizer/get_config. Slimmed hf_transformers_patches.py (torch-free only). patch_tokenizer.py verbatim
connector/ (stub) create_remote_connector raises NotImplementedError
Top-level slimmed files environ.py (env var registry) · server_args.py (device/revision/tokenizer_backend + PortArgs.init_new) · srt/utils/common.py (ImageData/VideoData/read_system_prompt_from_file + hf helpers) · sglang/utils.py (convert_json_schema_to_str/is_in_ci/TypeBasedDispatcher/LazyImport)

What's NOT included

Inference engine code is intentionally stripped: schedulers, model loaders, layer implementations, CUDA/Triton kernels, sampling, constrained decoding, speculative decoding, LoRA runtime, distributed runtime, KV cache manager, HTTP server, multimodal processing, observability, and all CLI/launch scripts.

License

Apache 2.0, adapted from sglang.

Project links

Data verified by PyPI on Aug 12, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for wxsms from gravatar.com wxsms

Credits

Author: wxsms

GitHub Statistics

Data verified by PyPI on Aug 12, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License (Apache License)

Requires

Python >=3.10

Provides Extra

fastokens test

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

llm_router_utils-0.2.0.tar.gz (231.2 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

llm_router_utils-0.2.0-py3-none-any.whl (290.0 kB view details)

Uploaded Python 3

File details

Details for the file llm_router_utils-0.2.0.tar.gz.

File metadata

  • Download URL: llm_router_utils-0.2.0.tar.gz
  • Upload date:
  • Size: 231.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for llm_router_utils-0.2.0.tar.gz
Algorithm Hash digest
SHA256 14e24d9af507af66c7b330ed62275ecb74bb4121cf8da5f253d782f32170a978
MD5 a05727f2df8a471c59ba82a9d20185a1
BLAKE2b-256 c596476a77e69e9989d3fd6a72dbee82186c7a06d83536381b501ef3fdbcdfcd

See more details on using hashes here.

Provenance

The following attestation bundles were made for llm_router_utils-0.2.0.tar.gz:

Publisher: publish.yml on wxsms/llm-router-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llm_router_utils-0.2.0-py3-none-any.whl.

File metadata

File hashes

Hashes for llm_router_utils-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 26d41ee5392b7712499af565c3c4632d032ce83261c129848f0209d949c2ae17
MD5 469127f8fc019ebc145022b8655c3ca7
BLAKE2b-256 009a585a63c502847577a4a0ea70b205706e063ede7e3d0bed3a10a780c3ca98

See more details on using hashes here.

Provenance

The following attestation bundles were made for llm_router_utils-0.2.0-py3-none-any.whl:

Publisher: publish.yml on wxsms/llm-router-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page