pylet 0.1.0

A simple distributed task execution system for GPU servers

PyLet

A simple distributed task execution system for GPU servers. Like Ray/K8s but much simpler.

Install

pip install pylet

For development:

git clone https://github.com/ServerlessLLM/pylet.git
cd pylet
pip install -e ".[dev]"

Quick Start

# Terminal 1: Start head node
pylet start

# Terminal 2: Start worker node with GPUs
pylet start --head localhost:8000 --gpu-units 4

# Terminal 3: Submit an instance
pylet submit 'vllm serve Qwen/Qwen2.5-1.5B-Instruct --port $PORT' \
    --gpu-units 1 --name my-vllm

# Check status
pylet get-instance --name my-vllm

# Get endpoint for inference
pylet get-endpoint --name my-vllm
# Output: 192.168.1.10:15600

# View logs
pylet logs <instance-id>

# Cancel
pylet cancel <instance-id>

See examples/README.md for more detailed examples including vLLM and SGLang.

Commands

Command	Description
pylet start	Start head node
pylet start --head <ip:port> --gpu-units N	Start worker with N GPUs
pylet submit <cmd> --gpu-units N --name <name>	Submit instance
pylet get-instance --name <name>	Get instance status
pylet get-endpoint --name <name>	Get instance endpoint (host:port)
pylet logs <id>	View instance logs
pylet logs <id> --follow	Follow logs in real-time
pylet cancel <id>	Cancel instance
pylet list-workers	List registered workers

Key Features

• Simple: No containers, no complex configs. Just pylet start and pylet submit.
• GPU-aware: Automatic GPU allocation via CUDA_VISIBLE_DEVICES.
• Service discovery: Instances get a PORT env var; endpoint available via get-endpoint.
• Real-time logs: Stream logs from running instances.
• Graceful shutdown: SIGTERM with configurable grace period before SIGKILL.

Requirements

• Python 3.9+
• Linux (tested on Ubuntu)

License

Apache 2.0