CLI tool for launching and managing vllm model servers via SSH and tmux

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for adefful from gravatar.com adefful

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License (MIT)
  • Author: adefful46@gmail.com
  • Requires: Python <4.0, >=3.8
Classifiers
Report project as malware

Project description

vllmctl

A powerful CLI for launching, managing, and monitoring vLLM model servers on remote machines via SSH and tmux.

โš ๏ธ SSH Configuration Required

Many commands in vllmctl rely on your SSH configuration (~/.ssh/config).

  • Make sure all your remote servers are properly listed in your SSH config.
  • The tool will automatically discover and use these hosts for remote operations, port forwarding, and GPU monitoring.

Example SSH config entry:

Host myserver
    HostName myserver.example.com
    User myuser
    IdentityFile ~/.ssh/id_rsa

๐Ÿš€ Features

  • Launch vLLM servers on remote hosts in isolated tmux sessions
  • Automatic SSH tunneling for secure local API access
  • Real-time health checks and queue monitoring
  • List, attach, and kill tmux sessions for full process control
  • GPU utilization dashboard across your cluster
  • Flexible model/port/env selection per launch
  • Safe for production: no processes die on SSH disconnect

๐Ÿ“ฆ Installation

pip install -r requirements.txt
  • Requires Python 3.8+
  • Ensure tmux is installed on both local and remote machines
  • Passwordless SSH access is recommended

๐Ÿš Shell Autocompletion

To enable shell autocompletion for vllmctl, run:

vllmctl --install-completion

Or, to see the completion script for your shell:

vllmctl --show-completion

You can add the output to your shell profile (e.g., .bashrc, .zshrc) for persistent autocompletion.

๐Ÿ› ๏ธ Commands Overview

1. list_local

Show all local vLLM models (including forwarded ports).

vllmctl list-local

Sample Output:

โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Server   โ”ƒ Remote portโ”ƒ Local portโ”ƒ Status       โ”ƒ Model                                โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ -        โ”‚ -          โ”‚ 8000      โ”‚ Local launch โ”‚ Qwen/Qwen2.5-Coder-32B-Instruct      โ”‚
โ”‚ server1  โ”‚ 8000       โ”‚ 16100     โ”‚ Forwarded    โ”‚ Llama-2-13B-chat                     โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

2. list_remote

Show vLLM models running on all servers from your SSH config.

vllmctl list-remote [--host-regex <pattern>] [--remote-port <port>] [--debug]

3. auto_forward

Automatically forward ports with running models to your local machine.

vllmctl auto-forward [--host-regex <pattern>] [--remote-port <port>] [--local-range <start-end>] [--no-kill] [--debug]

4. tmux_forwards

Show all tmux-based SSH forwards and their status.

vllmctl tmux-forwards

5. vllm_queue_top

Real-time dashboard for vLLM queue status on all local ports (like nvtop for vLLM).

vllmctl vllm-queue-top

Sample Output:

Scanning ports for vLLM models... โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 100% 0:00:00
                                         \ vLLM Queue Status (refreshes every 1.0s)                                          
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Local Port โ”ƒ Model          โ”ƒ Waiting โ”ƒ Running โ”ƒ Wait graph             โ”ƒ Run graph               โ”ƒ Prompt TPT โ”ƒ Gen TPT โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ 16004      โ”‚ Qwen/Qwen3-32B โ”‚ 774     โ”‚ 226     โ”‚ โ–โ–โ–‚โ–‚โ–‚โ–‚โ–ƒโ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–„โ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–…โ€ฆ โ”‚ โ–ˆโ–‡โ–†โ–†โ–†โ–†โ–…โ–†โ–†โ–…โ–…โ–„โ–„โ–…โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ€ฆ โ”‚ -          โ”‚ -       โ”‚
โ”‚ 16101      โ”‚ Qwen/Qwen3-32B โ”‚ 774     โ”‚ 226     โ”‚ โ–โ–โ–‚โ–‚โ–โ–‚โ–ƒโ–โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–…โ€ฆ โ”‚ โ–ˆโ–‡โ–†โ–†โ–‡โ–†โ–…โ–‡โ–†โ–…โ–…โ–…โ–„โ–…โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ€ฆ โ”‚ -          โ”‚ -       โ”‚
โ”‚ 16102      โ”‚ Qwen/Qwen3-4B  โ”‚ 663     โ”‚ 113     โ”‚ โ–โ–โ–โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–„โ–„โ–„โ–„โ–…โ–…โ–„โ–…โ–…โ€ฆ โ”‚ โ–ˆโ–‡โ–‡โ–†โ–†โ–†โ–†โ–†โ–…โ–…โ–…โ–„โ–„โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ€ฆ โ”‚ -          โ”‚ -       โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

6. gpu_idle_top

Live GPU utilization and memory dashboard for all servers in your SSH config.

vllmctl gpu-idle-top --host-regex <pattern>

Sample Output:

Scanning GPU utilization on hosts... โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 100% 0:00:27
                            | GPU Idle Top (refreshes every 0.5s)                            
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ Host              โ”ƒ Util (%) โ”ƒ Util Graph                     โ”ƒ Mem (%) โ”ƒ Mem Graph โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ host-a            โ”‚ 0.0      โ”‚ โ–โ–                             โ”‚ 5.1     โ”‚ โ–โ–        โ”‚
โ”‚ host-b            โ”‚ 0.0      โ”‚ โ–โ–                             โ”‚ 94.4    โ”‚ โ–โ–        โ”‚
โ”‚ host-c            โ”‚ 0.0      โ”‚ โ–โ–                             โ”‚ 0.0     โ”‚ โ–โ–        โ”‚
โ”‚ host-d            โ”‚ 86.5     โ”‚                             โ–ˆโ– โ”‚ 90.9    โ”‚ โ–โ–        โ”‚
โ”‚ host-e            โ”‚ 89.0     โ”‚                             โ–ˆโ– โ”‚ 59.3    โ”‚ โ–โ–        โ”‚
โ”‚ host-f            โ”‚ 91.9     โ”‚                             โ–โ–ˆ โ”‚ 92.9    โ”‚ โ–โ–        โ”‚
โ”‚ host-g            โ”‚ 95.0     โ”‚                             โ–ˆโ– โ”‚ 93.3    โ”‚ โ–โ–        โ”‚
โ”‚ host-h            โ”‚ 97.4     โ”‚                             โ–โ–ˆ โ”‚ 52.3    โ”‚ โ–โ–        โ”‚
โ”‚ host-i            โ”‚ 100.0    โ”‚                             โ–โ–ˆ โ”‚ 91.9    โ”‚ โ–โ–        โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

7. launch

Launch a vLLM server on a remote host and set up a local SSH tunnel.

vllmctl launch --server <host> --model <model_name> [--conda-env <env>] [--remote-port <port>] [--local-range <range>] [--timeout <seconds>] [--lifetime <duration>]
  • --lifetime <duration>: Maximum lifetime for the vLLM process. Supports formats like 10m (minutes), 2h (hours), 1d (days), 30s (seconds). Example: 
    vllmctl launch --server myserver --model Qwen/Qwen3-4B --lifetime 2h
  • After the specified lifetime, the vLLM server will be automatically stopped on the remote server.
  • Only the vLLM process runs in tmux on the remote server; the SSH tunnel is managed locally without tmux.

8. Other Utilities

  • Attach to tmux session: 
    vllmctl attach-tmux <session_name>
  • Kill a tmux session: 
    vllmctl kill-tmux <session_name>
  • Clean up dead/unused tmux sessions: 
    vllmctl clean-tmux-forwards

๐Ÿ“ Best Practices

  • Always use tmux for remote process management
  • Use SSH keys for authentication
  • Monitor endpoints with health checks and logs
  • Clean up unused sessions regularly
  • For production, consider systemd for static deployments

โ„น๏ธ Help

All commands support --help for detailed usage:

vllmctl <command> --help

License

MIT

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for adefful from gravatar.com adefful

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License (MIT)
  • Author: adefful46@gmail.com
  • Requires: Python <4.0, >=3.8
Classifiers

Release history Release notifications | RSS feed

0.2.1

0.2.0

This version

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

vllmctl-0.1.0.tar.gz (13.5 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

vllmctl-0.1.0-py3-none-any.whl (14.8 kB view details)

Uploaded Python 3

File details

Details for the file vllmctl-0.1.0.tar.gz.

File metadata

  • Download URL: vllmctl-0.1.0.tar.gz
  • Upload date:
  • Size: 13.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.9.21

File hashes

Hashes for vllmctl-0.1.0.tar.gz
Algorithm Hash digest
SHA256 920007dd7e94373a637ff3d238901571bc34d969d5a132207ac985939adcc0d5
MD5 d07f713b626ff65b095a83abb744741b
BLAKE2b-256 420eecd7724778e58fb656967cd82ce854d20089559bd25047b7830097da074b

See more details on using hashes here.

File details

Details for the file vllmctl-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: vllmctl-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 14.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.9.21

File hashes

Hashes for vllmctl-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 8c8823c00d3eb5128453ebc66dae3b9d3a17aed13e1875e150da85b43ad4b381
MD5 f04eb60214b2655df064821eeedb3a98
BLAKE2b-256 29859e40b909f5db9a46488db3b5e4c689d2b2a47e28f1ada35b38c7af806ae1

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page