llm-router-utils

Lightweight extraction of sglang's reasoning parser, tool-call parser, and chat template rendering, for use in custom router services and lightweight LLM applications.

This library does not include any inference engine code. It only provides the "frontend" message processing pipeline: OpenAIServingChat._process_messages and its dependencies.

Upstream source: sglang release/v0.5.16.

Installation

pip install llm-router-utils

Usage

from llm_router_utils.sglang.srt.configs.model_config import ModelConfig
from llm_router_utils.sglang.srt.managers.tokenizer_manager import TokenizerManager
from llm_router_utils.sglang.srt.parser.template_manager import TemplateManager
from llm_router_utils.sglang.srt.entrypoints.openai.serving_chat import OpenAIServingChat
from llm_router_utils.sglang.srt.server_args import ServerArgs, PortArgs

server_args = ServerArgs(model_path="Qwen/Qwen3-32B", tool_call_parser="qwen3_coder")
port_args = PortArgs()
tokenizer_manager = TokenizerManager(server_args, port_args)
template_manager = TemplateManager()
template_manager.initialize_templates(
    tokenizer_manager=tokenizer_manager,
    model_path=server_args.model_path,
    chat_template=server_args.chat_template,
)
openai_serving_chat = OpenAIServingChat(tokenizer_manager, template_manager)
result = openai_serving_chat._process_messages(request, is_multimodal=False)

What's included

Migrated modules under llm_router_utils/sglang/srt/ (paths preserved from upstream):

Module Description
parser/ Conversation templates (~50 model families), ReasoningParser + detectors, harmony/inkling, jinja utils, template detection, TemplateManager with TokenizerLike Protocol
function_call/ FunctionCallParser + 33 detectors (hermes, glm, deepseek, qwen, kimi, mistral, …), JsonArrayParser, schema utils
entrypoints/openai/ protocol.py (~1900 lines, full OpenAI types), slimmed serving_chat.py (only _process_messages chain), serving_base.py, chat_encoding.py, encoding_dsv32/dsv4.py, sse_utils.py, usage_processor.py, utils.py
managers/ Slimmed TokenizerManager (uses upstream get_tokenizer for byte-parity incl. SGLANG_PATCH_TOKENIZER), slimmed io_struct.py, embed_types.py stub
configs/ Slimmed ModelConfig (uses upstream get_config; exposes hf_config/is_multimodal/get_default_sampling_params/context_length), model_config_parser_registry.py
tokenizer/ tiktoken_tokenizer.py
disaggregation/ kv_events.py — KV cache event structs (EventBatch, KVCacheEvent, StorageMedium, BlockStored, BlockRemoved, AllBlocksCleared, KVEventBatch)
mem_cache/ utils.py — pure-Python SHA256 hash helpers, byte-identical to sglang's C++ extension
observability/ metrics_collector.py — data classes only: QueueCount, SchedulerStats, compute_routing_key_stats. Heavy *MetricsCollector classes stripped
utils/hf_transformers/ Restored common.py/config.py/tokenizer.py/mistral_utils.py — upstream get_tokenizer/get_config. Slimmed hf_transformers_patches.py (torch-free only). patch_tokenizer.py verbatim
connector/ (stub) create_remote_connector raises NotImplementedError
Top-level slimmed files environ.py (env var registry) · server_args.py (device/revision/tokenizer_backend + PortArgs.init_new) · srt/utils/common.py (ImageData/VideoData/read_system_prompt_from_file + hf helpers) · sglang/utils.py (convert_json_schema_to_str/is_in_ci/TypeBasedDispatcher/LazyImport)

What's NOT included

Inference engine code is intentionally stripped: schedulers, model loaders, layer implementations, CUDA/Triton kernels, sampling, constrained decoding, speculative decoding, LoRA runtime, distributed runtime, KV cache manager, HTTP server, multimodal processing, observability, and all CLI/launch scripts.

License

Apache 2.0, adapted from sglang.

Project links

Data verified by PyPI on Aug 12, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for wxsms from gravatar.com wxsms

Credits

Author: wxsms

GitHub Statistics

Data verified by PyPI on Aug 12, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License (Apache License)

Requires

Python >=3.10

Provides Extra

fastokens test

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

llm_router_utils-0.2.1.tar.gz (231.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

llm_router_utils-0.2.1-py3-none-any.whl (290.0 kB view details)

Uploaded Python 3

File details

Details for the file llm_router_utils-0.2.1.tar.gz.

File metadata

  • Download URL: llm_router_utils-0.2.1.tar.gz
  • Upload date:
  • Size: 231.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for llm_router_utils-0.2.1.tar.gz
Algorithm Hash digest
SHA256 6fd017741f193ae798a32b34c540b124e5aa73376ae4788b3a2fc659ad0077cc
MD5 7e18837a4a2f9dae3a480b8f84736c4f
BLAKE2b-256 92d600a3ead3113d79ebdeff4fc31972f579db215b2c6c3f1e87cb6e223477d5

See more details on using hashes here.

Provenance

The following attestation bundles were made for llm_router_utils-0.2.1.tar.gz:

Publisher: publish.yml on wxsms/llm-router-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llm_router_utils-0.2.1-py3-none-any.whl.

File metadata

File hashes

Hashes for llm_router_utils-0.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 dea7e428388329df443bf25434bf2a66241678e573cc7878fff24d40acad23b3
MD5 fd13d775f40f2325791c91cc6267451c
BLAKE2b-256 1080db95d200ef47de5ce8f814d51b09989c5294108473e19cdb8abeab148bd4

See more details on using hashes here.

Provenance

The following attestation bundles were made for llm_router_utils-0.2.1-py3-none-any.whl:

Publisher: publish.yml on wxsms/llm-router-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page