4 projects
smg
High-performance Rust-based inference gateway for large-scale LLM deployments
smg-grpc-servicer
SMG gRPC servicer implementations for LLM inference engines (vLLM, SGLang)
smg-grpc-proto
SMG gRPC proto definitions for SGLang, vLLM, and TRT-LLM
genai-bench
A powerful benchmark tool designed for comprehensive token-level performance evaluation of large language model (LLM) serving systems.
