LLM Benchmark Studio

Production-grade, local-first performance evaluation suite for OpenAI-compatible LLM endpoints.

LLM Benchmark Studio allows you to easily evaluate, benchmark, and visualize the throughput, latency, and token-level performance of any OpenAI-compatible API endpoint (including vLLM, LM Studio, Ollama, SGLang, TGI, OpenAI, and local custom routers).

Key Features

• 🔌 Connections Manager: Configure and store endpoint connections, target models, temperature settings, and custom request headers with a built-in lightweight network connection and latency tester.
• 📝 Prompt Dataset Editor: Import prompt datasets (CSV/JSON), customize individual prompts, and estimate token counts using client-side tokenizer heuristics.
• ⚡ Performance Benchmarking Engine: Run concurrent evaluations using customized batch sizes and warmup requests. Calculate throughput (Tokens/sec), Time to First Token (TTFT), and Inter-Token Latency (ITL).
• 🎛️ Force Max Tokens (Ignore EOS): Optional switch to ignore instruct model End-of-Sequence (EOS) tokens, forcing models to generate exactly the requested output token length.
• 🔎 Server-Side Token Verification: Requests use "stream_options": {"include_usage": true} to extract exact, server-reported input/output token counts for absolute timing accuracy.
• 📊 Parameter Sweep Matrix: Benchmark multiple parameter combinations (concurrencies, token sizes) in sequence. Generates sweep combinations automatically and visualizes performance frontiers.
• 📈 Real-Time Charts & Dashboard: Live progress tracking and interactive Apache ECharts displaying latency distributions and tokens/second throughput.
• 📄 Executive Reports: Export comparative benchmark reports side-by-side as print-ready HTML, compiled PDF documents (using WeasyPrint with a structured ReportLab fallback), or detailed multi-sheet Excel workbooks.

Installation

Install LLM Benchmark Studio using pip:

pip install llm-bench-studio

Note: Requires Python 3.10 or higher.

Quick Start

Launch both the user interface and backend server using a single command:

llm-bench-studio start

Once running, open your browser and navigate to: 👉 http://localhost:8005

Custom Host and Port Configuration

To bind the server to a custom interface or port, pass options to the CLI:

llm-bench-studio start --host 0.0.0.0 --port 8080

Local Development Setup

If you wish to run LLM Benchmark Studio from source or contribute changes:

1. Clone the Repository:

   git clone https://github.com/bezawadasiddinikhilesh/LLM_Bench_Studio.git
   cd LLM_Bench_Studio
   

2. Backend Setup:

   # Create a virtual environment
   python3 -m venv venv
   source venv/bin/activate
   
   # Install dependencies
   pip install -r backend/requirements.txt
   
   # Start the API server in reload mode
   python3 -m uvicorn backend.main:app --port 8005 --reload
   

3. Frontend Setup:

   # Install node dependencies
   npm install
   
   # Run Vite development server
   npm run dev
   

4. Compiling a Package: Compile Vite assets and build a wheel distribution package locally:

   ./build_package.sh
   

License

This project is licensed under the MIT License.