min-llm-server-client 0.4.3

A minimal API server for local HuggingFace LLMs or VLLM LLMs

Minimal LLM Server, for API calls

The simplest possible Python code for running local LLM inference as a REST API server and a simple client.

This package lets you start an inference server for Hugging Face–compatible models (like LLaMA, Qwen, GPT-OSS, etc.) on your own computer or server, and make it accessible to applications via HTTP. It supports both standard HuggingFace Transformers and high-performance vLLM backends.

See the Tutorial page for extented info.

Backend Options

This package now supports two inference backends:

1. HuggingFace Transformers (Standard)

• ✓ Widely compatible
• ✓ CPU support available
• ✓ Smaller installation size
• ✓ Good for development and testing

2. vLLM Optimized (High-Performance)

• ✓ Up to 24x faster throughput than standard transformers
• ✓ Lower latency for single requests
• ✓ Better GPU memory utilization with PagedAttention
• ✓ Automatic multi-GPU support with tensor parallelism
• ✓ Continuous batching for higher throughput
• ⚠ Requires CUDA GPUs (no CPU support)
• ⚠ Best for production deployments

---

Installation by pip

Prerequisite

uv venv --python 3.12
source .venv/bin/activate

Standard light weight Installation (HuggingFace):

uv pip install min-llm-server-client

With vLLM Support:

uv pip install "min-llm-server-client[vllm]"

Installation From Source:

git clone https://github.com/afshinsadeghi/min_llm_server_client.git
cd min_llm_server_client

# Standard installation
uv pip install .

# Or with vLLM support
uv pip install ".[vllm]"

---

Usage

Starting the Server

Standard HuggingFace Transformers Server

uv run min-llm-server --model_name meta-llama/Llama-3.3-70B-Instruct --max_new_tokens 100 --device cuda:0

vLLM Optimized infernce Server

uv run min-llm-server-vllm --model_name openai/gpt-oss-20b --max_new_tokens 100 --device cuda:2

Command Options:

• --model_name : Hugging Face model name or local path suggested models: openai/gpt-oss-20b openai/gpt-oss-120b meta-llama/Llama-3.3-70B-Instruct meta-llama/Llama-3.1-8B Qwen/Qwen3-0.6B Qwen/Qwen2-VL-72B-Instruct-AWQ deepseek-ai/DeepSeek-R1-Distill-Qwen-32B

  or it can use a local model on your device with /path/to/model.

• --max_new_tokens : maximum number of tokens to generate in response.

• --device : Device selection

  • auto - Auto-detect available GPUs (default)
  • cpu, - Force CPU (HuggingFace only, vLLM requires GPU)
  • cuda:0, cuda:1 , or a list of GPU cores: cuda:2,3,4,5,6,7.

• Specific to vLLM : --max_model_len : Maximum model context length. If not specified, will auto-detect from model config. Example: 8192

If the device parameter is not given or is auto, it finds the available GPU cores and uses them and if no gpu is available, it uses CPU instead.

Example run:

Standard server with default settings (auto GPU detection):

min-llm-server 

Standard server on a specific GPU (e.g., GPU 0):

min-llm-server --model_name openai/gpt-oss-20b --device cuda:0

Standard server on a specific GPU (e.g., GPU 1):

min-llm-server --model_name openai/gpt-oss-120b --device cuda:1

Standard server forced on CPU:

min-llm-server --model_name openai/gpt-oss-20b --max_new_tokens 50 --device cpu

vLLM server with auto GPU detection (uses all available GPUs):

min-llm-server-vllm --model_name meta-llama/Llama-3.3-70B-Instruct

vLLM server on a specific GPU (e.g., GPU 2):

min-llm-server-vllm --model_name meta-llama/Llama-3.3-70B-Instruct --device cuda:2

Standard server on a several GPUs:

min-llm-server --model_name meta-llama/Llama-3.3-70B-Instruct --device cuda:2,3,4,5,6,7

---

Sending Queries

Once the server is running (default: http://127.0.0.1:5000/llm/q), you can query it with curl or Python.

Curl:

curl -X POST http://127.0.0.1:5000/llm/q \
  -H "Content-Type: application/json" \
  -d '{"query": "What is Earth?", "key": "key1"}'

Python client:

from min_llm_server_client.local_llm_inference_api_client import send_query

response = send_query("What is the capital of France?", user="user1", key="key1")
print(response)

---

Performance Comparison

LLaMA 3.1 8B - Standard HuggingFace Backend:

• Intel CPU → ~30 seconds per request, ~2.4 GB RAM
• A100 GPU → <1 second per request, ~34 GB GPU memory, ~4.8 GB CPU RAM

LLaMA 3.1 8B - vLLM Optimized Backend:

• A100 GPU → ~0.1-0.3 seconds per request (3-10x faster)
• Better memory efficiency with PagedAttention
• Supports higher concurrent request throughput

Performance Tips:

• Use vLLM for production deployments with high request volumes
• Use standard backend for development, testing, or CPU-only environments
• Both the deployement method based on Hugging face and vLLM automatically utilize multiple GPUs, vLLM with tensor parallelism
• Both backends support the same API, making it easy to switch

---

Project Structure

min_llm_server_client/
├── src/
│   ├── local_llm_inference_api_client.py
│   ├── local_llm_inference_server_api.py
│   └── ...
└── README.md

---

License

This project is open source under the Apache 2.0 License.

---

Author

Afshin Sadeghi
🔗 GitHub
🔗 Google Scholar
🔗 LinkedIn