llm-router-utils

Lightweight extraction of sglang's reasoning parser, tool-call parser, and chat template rendering, for use in custom router services and lightweight LLM applications.

This library does not include any inference engine code. It only provides the "frontend" message processing pipeline: OpenAIServingChat._process_messages and its dependencies.

Upstream source: sglang release/v0.5.16.

Installation

pip install llm-router-utils

Usage

from llm_router_utils.sglang.srt.configs.model_config import ModelConfig
from llm_router_utils.sglang.srt.managers.tokenizer_manager import TokenizerManager
from llm_router_utils.sglang.srt.parser.template_manager import TemplateManager
from llm_router_utils.sglang.srt.entrypoints.openai.serving_chat import OpenAIServingChat
from llm_router_utils.sglang.srt.server_args import ServerArgs, PortArgs

server_args = ServerArgs(model_path="Qwen/Qwen3-32B", tool_call_parser="qwen3_coder")
port_args = PortArgs()
tokenizer_manager = TokenizerManager(server_args, port_args)
template_manager = TemplateManager()
template_manager.initialize_templates(
    tokenizer_manager=tokenizer_manager,
    model_path=server_args.model_path,
    chat_template=server_args.chat_template,
)
openai_serving_chat = OpenAIServingChat(tokenizer_manager, template_manager)
result = openai_serving_chat._process_messages(request, is_multimodal=False)

What's included

Migrated modules under llm_router_utils/sglang/srt/ (paths preserved from upstream):

Module Description
parser/ Conversation templates (~50 model families), ReasoningParser + detectors, harmony/inkling, jinja utils, template detection, TemplateManager with TokenizerLike Protocol
function_call/ FunctionCallParser + 33 detectors (hermes, glm, deepseek, qwen, kimi, mistral, …), JsonArrayParser, schema utils
entrypoints/openai/ protocol.py (~1900 lines, full OpenAI types), slimmed serving_chat.py (only _process_messages chain), serving_base.py, chat_encoding.py, encoding_dsv32/dsv4.py, sse_utils.py, usage_processor.py, utils.py
managers/ Slimmed TokenizerManager (uses upstream get_tokenizer for byte-parity incl. SGLANG_PATCH_TOKENIZER), slimmed io_struct.py, embed_types.py stub
configs/ Slimmed ModelConfig (uses upstream get_config; exposes hf_config/is_multimodal/get_default_sampling_params/context_length), model_config_parser_registry.py
tokenizer/ tiktoken_tokenizer.py
disaggregation/ kv_events.py — KV cache event structs (EventBatch, KVCacheEvent, StorageMedium, BlockStored, BlockRemoved, AllBlocksCleared, KVEventBatch)
mem_cache/ utils.py — pure-Python SHA256 hash helpers, byte-identical to sglang's C++ extension
observability/ metrics_collector.py — data classes only: QueueCount, SchedulerStats, compute_routing_key_stats. Heavy *MetricsCollector classes stripped
utils/hf_transformers/ Restored common.py/config.py/tokenizer.py/mistral_utils.py — upstream get_tokenizer/get_config. Slimmed hf_transformers_patches.py (torch-free only). patch_tokenizer.py verbatim
connector/ (stub) create_remote_connector raises NotImplementedError
Top-level slimmed files environ.py (env var registry) · server_args.py (device/revision/tokenizer_backend + PortArgs.init_new) · srt/utils/common.py (ImageData/VideoData/read_system_prompt_from_file + hf helpers) · sglang/utils.py (convert_json_schema_to_str/is_in_ci/TypeBasedDispatcher/LazyImport)

What's NOT included

Inference engine code is intentionally stripped: schedulers, model loaders, layer implementations, CUDA/Triton kernels, sampling, constrained decoding, speculative decoding, LoRA runtime, distributed runtime, KV cache manager, HTTP server, multimodal processing, observability, and all CLI/launch scripts.

License

Apache 2.0, adapted from sglang.

Project links

Data verified by PyPI on Aug 13, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for wxsms from gravatar.com wxsms

Credits

Author: wxsms

GitHub Statistics

Data verified by PyPI on Aug 13, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License (Apache License)

Requires

Python >=3.10

Provides Extra

fastokens test

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

llm_router_utils-0.2.2.tar.gz (234.1 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

llm_router_utils-0.2.2-py3-none-any.whl (293.0 kB view details)

Uploaded Python 3

File details

Details for the file llm_router_utils-0.2.2.tar.gz.

File metadata

  • Download URL: llm_router_utils-0.2.2.tar.gz
  • Upload date:
  • Size: 234.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for llm_router_utils-0.2.2.tar.gz
Algorithm Hash digest
SHA256 c62adc1b1e1d9fe7742edec3edc37260a230666ee0585898e4a8f1ff8de46f2f
MD5 7a980817a6595d2f3b7a0a13170d81b6
BLAKE2b-256 91ef3244f309f7d9e0f3674d7d062931b344cc2141684f473f82831deae14d5b

See more details on using hashes here.

Provenance

The following attestation bundles were made for llm_router_utils-0.2.2.tar.gz:

Publisher: publish.yml on wxsms/llm-router-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llm_router_utils-0.2.2-py3-none-any.whl.

File metadata

File hashes

Hashes for llm_router_utils-0.2.2-py3-none-any.whl
Algorithm Hash digest
SHA256 27857e07c79db013381642b5ac30b150593f12b94a14326c49975799b7bf132e
MD5 a082f57d7a9961af8d052343ed6a45a9
BLAKE2b-256 5db0ac0cdb2be8e41d5c26791dbeda59ab802b4b149aaa33899464008216c1ed

See more details on using hashes here.

Provenance

The following attestation bundles were made for llm_router_utils-0.2.2-py3-none-any.whl:

Publisher: publish.yml on wxsms/llm-router-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page