llm-host

OpenAI-compatible inference server that runs an LLM (via vLLM), Whisper transcription/translation, and Kokoro TTS on a GPU and exposes them all at a single URL — optionally via ngrok for a public endpoint.

Default model: Qwen/Qwen3.5-2B + whisper-small — tuned to run on a T4 GPU (15 GB VRAM). Swap to larger models via --model / --whisper-model or env vars.

Designed for Google Colab (T4 / L4 / A100) but works on any GPU machine with CUDA.

Install

pip install llm-host

# vLLM must be installed separately (GPU/CUDA-specific build)
pip install "vllm>=0.6.0"

# Kokoro TTS requires espeak-ng for phonemization
apt-get install -y espeak-ng   # Debian/Ubuntu/Colab

Quickstart

With ngrok (public URL):

llm-host \
  --ngrok-token YOUR_NGROK_TOKEN \
  --hf-token    YOUR_HF_TOKEN

Without ngrok (localhost / LAN only):

llm-host --hf-token YOUR_HF_TOKEN
# accessible at http://localhost:5001  and  http://<server-ip>:5001

Or with environment variables:

NGROK_TOKEN=xxx HF_TOKEN=xxx llm-host

Without --ngrok-token the server binds to 0.0.0.0 and prints both the localhost and network IP URLs. Pass --ngrok-token to get a public ngrok URL.

GPU / CUDA notes

• CTranslate2 is pinned to <4.6. Version 4.8.x crashes on newer GPUs (H100/H200) with parallel_for failed: cudaErrorInvalidDevice; faster-whisper would otherwise pull it. The pin is handled for you by pip install llm-host.
• Whisper (CTranslate2) and Kokoro (PyTorch) both use cuDNN. If your environment resolves them to mismatched cuDNN builds you may see CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH on whichever loads second. On a clean Colab runtime the pre-installed CUDA stack avoids this; on a custom box, install a single consistent torch (and let it own cuDNN) before the extras.

Endpoints

All API endpoints are OpenAI-compatible — drop in the ngrok URL as base_url with any OpenAI SDK.

from openai import OpenAI

client = OpenAI(
    base_url="https://<ngrok-id>.ngrok-free.app/v1",
    api_key="dummy",
)

# Chat  (model name = --served-model-name, default is last part of --model)
# Default: "Qwen3.5-2B" — check active name via GET /v1/models
resp = client.chat.completions.create(
    model="Qwen3.5-2B",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Transcription
with open("audio.wav", "rb") as f:
    text = client.audio.transcriptions.create(model="whisper-1", file=f)

# TTS
client.audio.speech.create(
    model="tts-1", input="Hello!", voice="nova"
).stream_to_file("out.mp3")

Model configuration

LLM (reasoning / chat)

Set via --model or MODEL= env var. Default is Qwen/Qwen3.5-2B — runs on T4, no HuggingFace token required.

# Default — T4-friendly, no HF token needed
llm-host

# Larger Qwen3 variants (A100 recommended)
MODEL=Qwen/Qwen3-8B  llm-host
MODEL=Qwen/Qwen3-14B llm-host
MODEL=Qwen/Qwen3-32B llm-host

# Llama 3.1 (gated — requires HF token + accepted licence)
MODEL=meta-llama/Llama-3.1-8B-Instruct HF_TOKEN=hf_xxx llm-host

# AWQ-quantized Llama (lower VRAM, still needs A100 for large-v3 Whisper)
MODEL=hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 QUANTIZATION=awq llm-host

The model is served under its last path component by default (e.g. Qwen3.5-2B). Override with --served-model-name / SERVED_MODEL_NAME=.

Whisper (speech-to-text)

Set via --whisper-model or WHISPER_MODEL= env var. Default: small (T4-friendly); use large-v3 for best accuracy.

WHISPER_MODEL=small       llm-host   # default — T4-friendly (~1 GB VRAM)
WHISPER_MODEL=medium      llm-host   # better accuracy, ~2 GB VRAM
WHISPER_MODEL=large-v3    llm-host   # highest accuracy, ~10 GB VRAM (A100)
WHISPER_MODEL=large-v3-turbo llm-host  # good balance on A100

CLI options

llm-host --help

  --ngrok-token              ngrok authtoken (optional; omit for localhost/LAN only)
  --hf-token                 HuggingFace token (needed only for gated models)
  --model                    HuggingFace model ID (default: Qwen/Qwen3.5-2B)
  --served-model-name        name used in API calls (default: last part of --model)
  --quantization             awq | bitsandbytes | none  (default: none)
  --whisper-model            tiny | base | small | medium | large-v1 | large-v2 | large-v3 | large-v3-turbo
                             (default: small)
  --tts-voice                alloy | echo | fable | onyx | nova | shimmer  (default: alloy)
  --vllm-port                internal vLLM port (default: 8000)
  --gateway-port             public gateway port (default: 5001)
  --gpu-memory-utilization   vLLM GPU memory fraction (default: 0.75).
                             vLLM pre-allocates this share of VRAM for model + KV cache +
                             CUDA graphs (vLLM ≥0.21 includes CUDA graph memory here).
                             0.75 works on T4 for 2B models; use ~0.85 for 8B+ on A100.
  --max-model-len            context length (default: 8192)
  --no-vllm                  skip starting vLLM (use existing instance)

All flags can also be set via UPPER_SNAKE_CASE environment variables:

MODEL=Qwen/Qwen3-14B \
WHISPER_MODEL=large-v3-turbo \
NGROK_TOKEN=xxx \
llm-host

TTS voices

Raw Kokoro voice names (e.g. af_sky) are also accepted directly.

License

MIT

Contact

For issues, questions, or feedback, reach out at saurabh.kpxt@gmail.com.