llm-host
OpenAI-compatible inference server that runs an LLM (via vLLM), Whisper transcription/translation, and Kokoro TTS on a GPU and exposes them all at a single URL — optionally via ngrok for a public endpoint.
Default model: Qwen/Qwen3.5-2B + whisper-small — tuned to run on a T4 GPU (15 GB VRAM). Swap to larger models via --model / --whisper-model or env vars.
Designed for Google Colab (T4 / L4 / A100) but works on any GPU machine with CUDA.
Install
pip install llm-host
# vLLM must be installed separately (GPU/CUDA-specific build)
pip install "vllm>=0.6.0"
# Kokoro TTS requires espeak-ng for phonemization
apt-get install -y espeak-ng # Debian/Ubuntu/Colab
Quickstart
With ngrok (public URL):
llm-host \
--ngrok-token YOUR_NGROK_TOKEN \
--hf-token YOUR_HF_TOKEN
Without ngrok (localhost / LAN only):
llm-host --hf-token YOUR_HF_TOKEN
# accessible at http://localhost:5001 and http://<server-ip>:5001
Or with environment variables:
NGROK_TOKEN=xxx HF_TOKEN=xxx llm-host
Without --ngrok-token the server binds to 0.0.0.0 and prints both the
localhost and network IP URLs. Pass --ngrok-token to get a public ngrok URL.
GPU / CUDA notes
- CTranslate2 is pinned to
<4.6. Version 4.8.x crashes on newer GPUs (H100/H200) withparallel_for failed: cudaErrorInvalidDevice;faster-whisperwould otherwise pull it. The pin is handled for you bypip install llm-host. - Whisper (CTranslate2) and Kokoro (PyTorch) both use cuDNN. If your
environment resolves them to mismatched cuDNN builds you may see
CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCHon whichever loads second. On a clean Colab runtime the pre-installed CUDA stack avoids this; on a custom box, install a single consistenttorch(and let it own cuDNN) before the extras.
Endpoints
| Method | Path | Description |
|---|---|---|
GET |
/ |
Dashboard UI |
GET |
/health |
Service status |
GET |
/v1/models |
List models |
POST |
/v1/chat/completions |
LLM chat (streaming supported) |
POST |
/v1/audio/transcriptions |
Whisper STT (keep source language) |
POST |
/v1/audio/translations |
Whisper STT → English |
POST |
/v1/audio/speech |
Kokoro TTS |
All API endpoints are OpenAI-compatible — drop in the ngrok URL as base_url with any OpenAI SDK.
from openai import OpenAI
client = OpenAI(
base_url="https://<ngrok-id>.ngrok-free.app/v1",
api_key="dummy",
)
# Chat (model name = --served-model-name, default is last part of --model)
# Default: "Qwen3.5-2B" — check active name via GET /v1/models
resp = client.chat.completions.create(
model="Qwen3.5-2B",
messages=[{"role": "user", "content": "Hello!"}],
)
# Transcription
with open("audio.wav", "rb") as f:
text = client.audio.transcriptions.create(model="whisper-1", file=f)
# TTS
client.audio.speech.create(
model="tts-1", input="Hello!", voice="nova"
).stream_to_file("out.mp3")
Model configuration
LLM (reasoning / chat)
Set via --model or MODEL= env var. Default is Qwen/Qwen3.5-2B — runs on T4, no HuggingFace token required.
# Default — T4-friendly, no HF token needed
llm-host
# Larger Qwen3 variants (A100 recommended)
MODEL=Qwen/Qwen3-8B llm-host
MODEL=Qwen/Qwen3-14B llm-host
MODEL=Qwen/Qwen3-32B llm-host
# Llama 3.1 (gated — requires HF token + accepted licence)
MODEL=meta-llama/Llama-3.1-8B-Instruct HF_TOKEN=hf_xxx llm-host
# AWQ-quantized Llama (lower VRAM, still needs A100 for large-v3 Whisper)
MODEL=hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 QUANTIZATION=awq llm-host
The model is served under its last path component by default (e.g. Qwen3.5-2B).
Override with --served-model-name / SERVED_MODEL_NAME=.
Whisper (speech-to-text)
Set via --whisper-model or WHISPER_MODEL= env var. Default: small (T4-friendly); use large-v3 for best accuracy.
| Model | VRAM | Speed | Accuracy |
|---|---|---|---|
tiny |
~1 GB | fastest | lowest |
base |
~1 GB | fast | low |
small |
~2 GB | fast | good |
medium |
~5 GB | moderate | better |
large-v2 |
~10 GB | slow | high |
large-v3 |
~10 GB | slow | highest |
large-v3-turbo |
~6 GB | fast | high |
WHISPER_MODEL=small llm-host # default — T4-friendly (~1 GB VRAM)
WHISPER_MODEL=medium llm-host # better accuracy, ~2 GB VRAM
WHISPER_MODEL=large-v3 llm-host # highest accuracy, ~10 GB VRAM (A100)
WHISPER_MODEL=large-v3-turbo llm-host # good balance on A100
CLI options
llm-host --help
--ngrok-token ngrok authtoken (optional; omit for localhost/LAN only)
--hf-token HuggingFace token (needed only for gated models)
--model HuggingFace model ID (default: Qwen/Qwen3.5-2B)
--served-model-name name used in API calls (default: last part of --model)
--quantization awq | bitsandbytes | none (default: none)
--whisper-model tiny | base | small | medium | large-v1 | large-v2 | large-v3 | large-v3-turbo
(default: small)
--tts-voice alloy | echo | fable | onyx | nova | shimmer (default: alloy)
--vllm-port internal vLLM port (default: 8000)
--gateway-port public gateway port (default: 5001)
--gpu-memory-utilization vLLM GPU memory fraction (default: 0.75).
vLLM pre-allocates this share of VRAM for model + KV cache +
CUDA graphs (vLLM ≥0.21 includes CUDA graph memory here).
0.75 works on T4 for 2B models; use ~0.85 for 8B+ on A100.
--max-model-len context length (default: 8192)
--no-vllm skip starting vLLM (use existing instance)
All flags can also be set via UPPER_SNAKE_CASE environment variables:
MODEL=Qwen/Qwen3-14B \
WHISPER_MODEL=large-v3-turbo \
NGROK_TOKEN=xxx \
llm-host
TTS voices
| Voice | Character | Kokoro name |
|---|---|---|
alloy |
Neutral female | af_heart |
echo |
Male | am_echo |
fable |
British female | bf_emma |
onyx |
Deep male | am_adam |
nova |
Energetic female | af_nova |
shimmer |
Soft female | af_bella |
Raw Kokoro voice names (e.g. af_sky) are also accepted directly.
License
MIT
Contact
For issues, questions, or feedback, reach out at saurabh.kpxt@gmail.com.
Metadata
Release files for llm-host 0.2.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_host-0.2.8.tar.gz | 21.1 kB | Without provenance | Details for llm_host-0.2.8.tar.gz |
Release files / llm_host-0.2.8.tar.gz
| Download URL | llm_host-0.2.8.tar.gz |
|---|---|
| Size | 21.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
58d1296d89af9b43ed8342e5f2ea3d03a593ff379b3853f63bf966acc8c8a65e
|
|
BLAKE2b-256 checksum How to use checksums |
47064c4427a9ad3d07a025ea475939bf7b22dadef1e43873ef9ee166f8f74570
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Security for llm-host 0.2.8
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.