min-llm-server-client 0.4.9

A minimal API server for local HuggingFace LLMs or VLLM LLMs

Minimal LLM Server, for API calls

The simplest possible Python code for running local LLM inference as a REST API server and a simple client.

This package lets you start an inference server for Hugging Face–compatible models (like LLaMA, Qwen, GPT-OSS, etc.) on your own computer or server, and make it accessible to applications via HTTP. It supports both standard HuggingFace Transformers and high-performance vLLM backends.

⚡ Optimized for maximum performance out of the box! The vLLM backend automatically enables prefix caching (2-10x speedup), CUDA graphs (10-30% speedup), and chunked prefill for optimal inference speed with zero configuration required.

See the Tutorial page for extented info.

Backend Options

This package now supports two inference backends:

1. HuggingFace Transformers (Standard)

• ✓ Widely compatible
• ✓ CPU support available
• ✓ Smaller installation size
• ✓ Good for development and testing

2. vLLM Optimized (High-Performance)

• ✓ Up to 24x faster throughput than standard transformers
• ✓ ⚡ Maximum performance out of the box - Prefix caching, CUDA graphs, and chunked prefill enabled by default
• ✓ Lower latency for single requests
• ✓ Better GPU memory utilization with PagedAttention
• ✓ Automatic multi-GPU support with tensor parallelism
• ✓ Continuous batching for higher throughput
• ✓ 🚀 Automatic optimization for quantized models - Detects GPTQ/AWQ/Int4 models and applies optimal vLLM parameters
• ⚠ Requires CUDA GPUs (no CPU support)
• ⚠ Best for production deployments

In comparison to the original vLLM min_llm_server_client:

• ✓ ⚡ Optimized for maximum performance out of the box - All performance features enabled by default
• ✓ Automatic GPU selection based on free VRAM
• ✓ Auto-configured multi-GPU tensor parallelism
• ✓ Smart quantized model detection - Automatically applies optimal vLLM parameters for quantized multi-GPU setups
• ✓ Ultra-lightweight API with minimal setup and dependencies, allows just setup and run with minimal or no configuration
• ✓ Easier to customize and integrate into research or internal AI pipelines in research clusters.

---

Installation by pip

Prerequisite

uv venv --python 3.12
source .venv/bin/activate

Standard light weight Installation (HuggingFace):

uv pip install min-llm-server-client

With vLLM Support:

uv pip install "min-llm-server-client[vllm]"

Installation From Source:

git clone https://github.com/afshinsadeghi/min_llm_server_client.git
cd min_llm_server_client

# Standard installation
uv pip install .

# Or with vLLM support
uv pip install ".[vllm]"

---

Usage

Starting the Server

Standard HuggingFace Transformers Server

uv run min-llm-server --model_name meta-llama/Llama-3.3-70B-Instruct --max_new_tokens 100 --device cuda:0

vLLM Optimized infernce Server

uv run min-llm-server-vllm --model_name openai/gpt-oss-20b --max_new_tokens 100 --device cuda:2

Command Options:

• --model_name : Hugging Face model name or local path suggested models: openai/gpt-oss-20b openai/gpt-oss-120b meta-llama/Llama-3.3-70B-Instruct
casperhansen/llama-3.3-70b-instruct-awq The VLLM version runs this only on ONE A100 core" meta-llama/Llama-3.1-8B Qwen/Qwen3-0.6B Qwen/Qwen2-VL-72B-Instruct-AWQ deepseek-ai/DeepSeek-R1-Distill-Qwen-32B Qwen/Qwen3-235B-A22B-FP8 Runs it with 4 A100 cores Qwen/Qwen3-30B-A3B-Instruct-250 Qwen/Qwen3.5-397B-A17B-GPTQ-Int4 Runs it with 4 A100 cores and optimzed setting for faster infernce or it can use a local model on your device with /path/to/model.

• --max_new_tokens : maximum number of tokens to generate in response.

• --device : Device selection

  • auto - Auto-detect available GPUs (default)
  • cpu, - Force CPU (HuggingFace only, vLLM requires GPU)
  • cuda:0, cuda:1 , or a list of GPU cores: cuda:2,3,4,5,6,7.

• Specific to vLLM :

  • --max_model_len : Maximum model context length. If not specified, will auto-detect from model config. Example: 8192
  • --gpu_memory_utilization : Fraction of GPU memory to use (0.0 to 1.0). Default: 0.90 (90%). Lower this value if sharing GPU with other processes. Examples: 0.85, 0.80, 0.75
  • --max_num_seqs : Maximum number of sequences to process in parallel. Lower this if you get 'max_num_seqs exceeds available cache blocks' errors. Examples: 256, 396, 512, 1024

  Performance Optimization Parameters (Enabled by Default):

  • --enable_prefix_caching : Enable prefix caching for faster inference with repeated prompts (default: True). Provides 2-10x speedup for queries with common prefixes like system prompts. Set to false to disable.
  • --enable_chunked_prefill : Enable chunked prefill for better batching with mixed sequence lengths (default: True). Improves throughput and reduces latency spikes. Set to false to disable.
  • --enforce_eager : Force eager execution mode, disabling CUDA graphs (default: False). CUDA graphs provide 10-30% speedup. Set to true only for debugging.

  Advanced Performance Parameters (Optional):

  • --swap_space : CPU swap space in GB for offloading during memory pressure (default: 4). Increase this to prevent OOM errors during traffic spikes. Example: --swap_space 16
  • --max_num_batched_tokens : Maximum number of tokens to batch together (default: auto). Controls throughput vs latency tradeoff. Example: --max_num_batched_tokens 4096

If the device parameter is not given or is auto, it finds the available GPU cores and uses them and if no gpu is available, it uses CPU instead.

Example run:

Standard server with default settings (auto GPU detection):

min-llm-server 

Standard server on a specific GPU (e.g., GPU 0):

min-llm-server --model_name openai/gpt-oss-20b --device cuda:0

Standard server on a specific GPU (e.g., GPU 1):

min-llm-server --model_name openai/gpt-oss-120b --device cuda:1

Standard server forced on CPU:

min-llm-server --model_name openai/gpt-oss-20b --max_new_tokens 50 --device cpu

vLLM server with auto GPU detection (uses all available GPUs):

min-llm-server-vllm --model_name meta-llama/Llama-3.3-70B-Instruct

vLLM server on a specific GPU (e.g., GPU 2):

min-llm-server-vllm --model_name meta-llama/Llama-3.3-70B-Instruct --device cuda:2

vLLM server with reduced GPU memory usage (for shared GPU scenarios):

min-llm-server-vllm --model_name meta-llama/Llama-3.3-70B-Instruct --device cuda:0 --gpu_memory_utilization 0.85

vLLM server with custom performance settings:

# Disable prefix caching (not recommended)
min-llm-server-vllm --model_name meta-llama/Llama-3.3-70B-Instruct --enable_prefix_caching false

# Increase swap space for stability under high load
min-llm-server-vllm --model_name meta-llama/Llama-3.3-70B-Instruct --swap_space 16

# Fine-tune batching for specific workload
min-llm-server-vllm --model_name meta-llama/Llama-3.3-70B-Instruct --max_num_batched_tokens 4096

Standard server on a several GPUs:

min-llm-server --model_name meta-llama/Llama-3.3-70B-Instruct --device cuda:2,3,4,5,6,7

---

Sending Queries

Once the server is running (default: http://127.0.0.1:5000/llm/q), you can query it with curl or Python.

Basic Curl Example:

curl -X POST http://127.0.0.1:5000/llm/q \
  -H "Content-Type: application/json" \
  -d '{"query": "What is Earth?", "key": "key1"}'

Advanced Curl Example with Generation Parameters:

curl -X POST http://127.0.0.1:5000/llm/q \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Explain quantum computing in simple terms",
    "key": "key1",
    "temperature": 0.7,
    "top_p": 0.95,
    "top_k": 50,
    "repetition_penalty": 1.1,
    "presence_penalty": 0.5
  }'

Python Client - Using LLMClient Class (Recommended):

from min_llm_server_client import LLMClient

# Initialize the client
client = LLMClient(base_url="http://127.0.0.1:5000", user_key="key1")

# Ask a question
answer = client.ask_question("What is the capital of France?")
print(answer)

Python Client - Advanced with Custom Parameters:

import requests
import json

url = "http://127.0.0.1:5000/llm/q"
payload = {
    "query": "Write a short poem about AI",
    "key": "key1",
    "temperature": 0.8,
    "top_p": 0.9,
    "top_k": 40,
    "repetition_penalty": 1.2,
    "presence_penalty": 0.6
}

response = requests.post(url, json=payload)
result = response.json()
print(result['answer'])

Available Generation Parameters:

• query (required): The text prompt/question
• key (required): API authentication key (default: "key1")
• temperature (optional, default: 0.1): Controls randomness (0.0 = deterministic, 1.0 = very random)
• top_p (optional, default: 0.9): Nucleus sampling threshold (0.0-1.0)
• top_k (optional, default: None): Top-k sampling - limits to k most likely tokens
• repetition_penalty (optional, default: 1.2): Penalty for repeating tokens (1.0 = no penalty)
• presence_penalty (optional, default: None): Penalty for using tokens that have appeared (vLLM only)
• extra_body (optional, default: None): Dictionary of additional custom parameters

Note: The server now handles requests asynchronously using a thread pool, preventing blocking during inference.

---

Performance Comparison

LLaMA 3.1 8B - Standard HuggingFace Backend:

• Intel CPU → ~30 seconds per request, ~2.4 GB RAM
• A100 GPU → <1 second per request, ~34 GB GPU memory, ~4.8 GB CPU RAM

LLaMA 3.1 8B - vLLM Optimized Backend:

• A100 GPU → ~0.1-0.3 seconds per request (3-10x faster)
• Better memory efficiency with PagedAttention
• Supports higher concurrent request throughput

Performance Tips:

• Use vLLM for production deployments with high request volumes
• Use standard backend for development, testing, or CPU-only environments
• Both the deployement method based on Hugging face and vLLM automatically utilize multiple GPUs, vLLM with tensor parallelism
• Both backends support the same API, making it easy to switch
• 🚀 Quantized models (GPTQ/AWQ/Int4) on multi-GPU setups are automatically optimized - No manual configuration needed!
• ⚡ Performance optimizations enabled by default - Prefix caching, chunked prefill, and CUDA graphs are automatically enabled for maximum speed

---

Automatic Performance Optimizations

The vLLM backend includes intelligent automatic optimizations that maximize inference speed without any manual configuration:

1. Default Performance Optimizations (Always Enabled)

These optimizations are enabled by default for all models and provide significant speedups:

• Prefix Caching (enable_prefix_caching=True)

  • Caches common prefixes like system prompts and chat templates
  • 2-10x speedup for queries with repeated prefixes
  • Zero overhead when prefixes don't repeat
  • Can be disabled with --enable_prefix_caching false if needed

• Chunked Prefill (enable_chunked_prefill=True)

  • Better batching for mixed sequence lengths
  • Improves throughput and reduces latency spikes
  • Handles long and short queries efficiently
  • Can be disabled with --enable_chunked_prefill false if needed

• CUDA Graphs (enforce_eager=False)

  • 10-30% speedup from CUDA graph optimization
  • Reduces kernel launch overhead
  • Enabled by default for all models
  • Can be disabled with --enforce_eager true for debugging only

2. Automatic Quantized Model Optimization

For quantized models (GPTQ/AWQ/Int4) running on multiple GPUs, additional optimizations are automatically applied:

How it works:

1. Automatic Detection: When you load a model, the server inspects the model's config.json to detect quantization (GPTQ, AWQ, Int4, etc.)
2. Multi-GPU Check: Determines if running on multiple GPUs via tensor parallelism
3. Smart Optimization: When both conditions are met, automatically applies:
   • kv_cache_dtype=fp8 - Uses FP8 for KV cache to maximize memory efficiency

Example - Running Qwen3.5-397B-A17B-GPTQ-Int4:

# Just run it - optimizations are applied automatically!
min-llm-server-vllm --model_name Qwen/Qwen3.5-397B-A17B-GPTQ-Int4 --device cuda:0,1,2,3,4,5,6,7

What you'll see:

CUDA graphs enabled (enforce_eager=False) for optimal performance
🚀 AUTOMATIC OPTIMIZATION ENABLED
Detected: Quantized model + Multi-GPU setup
Applying optimizations:
  • kv_cache_dtype = fp8   (Use FP8 for KV cache)

Benefits:

• ✓ Zero configuration - All optimizations work out of the box
• ✓ Maximum speed - Prefix caching (2-10x), CUDA graphs (10-30%), chunked prefill
• ✓ Better memory efficiency - FP8 KV cache saves VRAM for quantized models
• ✓ Optimal for all models - Automatic detection and optimization
• ✓ Transparent operation - Clear logging shows when optimizations are applied
• ✓ User control - All optimizations can be disabled via CLI if needed

Supported quantization formats:

• GPTQ (e.g., Qwen3.5-397B-A17B-GPTQ-Int4)
• AWQ (e.g., llama-3.3-70b-instruct-awq)
• Int4/Int8 quantization
• Any model with quantization_config or compression_config in its config

---

Project Structure

min_llm_server_client/
├── src/
│   ├── local_llm_inference_api_client.py
│   ├── local_llm_inference_server_api.py
│   └── ...
└── README.md

---

License

This project is open source under the Apache 2.0 License.

---

Author

Afshin Sadeghi
🔗 GitHub
🔗 Google Scholar
🔗 LinkedIn