A tool for submitting and managing distributed PyTorch jobs
Project description
Torch Submit
Introduction
Torch Submit is a lightweight, easy-to-use tool for running distributed PyTorch jobs across multiple machines. It's designed for researchers and developers who:
- Have access to a bunch of machines with IP addresses
- Want to run distributed PyTorch jobs without the hassle
- Don't have the time, energy, or patience to set up complex cluster management systems like SLURM or Kubernetes
Under the hood, Torch Submit uses Fabric to copy your working directory to the remote addresses and TorchRun to execute the command.
Features
- Simple cluster configuration: Just add your machines' IP addresses
- Easy job submission: Run your PyTorch jobs with a single command
- Job management: Submit, stop, restart, and monitor your jobs
- Log tailing: Easily view the logs of your running jobs
Installation
pip install torch-submit
or from source:
pip install -e . --prefix ~/.local
Quick Start
-
Set up a cluster:
torch-submit cluster create
Follow the interactive prompts to add your machines.
-
Submit a job:
torch-submit job submit --cluster my_cluster -- <entrypoint> # for example: # torch-submit job submit --cluster my_cluster -- python train.py # torch-submit job submit --cluster my_cluster -- python -m main.train
-
List running jobs:
torch-submit job list
-
Tail logs:
torch-submit logs tail <job_id>
-
Stop a job:
torch-submit job stop <job_id>
-
Restart a stopped job:
torch-submit job restart <job_id>
Usage
Cluster Management
- Create a cluster:
torch-submit cluster create
- List clusters:
torch-submit cluster list
- Remove a cluster:
torch-submit cluster remove <cluster_name>
Job Management
- Submit a job:
torch-submit job submit --cluster my_cluster -- <entrypoint>
- List jobs:
torch-submit job list
- Stop a job:
torch-submit job stop <job_id>
- Restart a job:
torch-submit job restart <job_id>
Log Management
- Tail logs:
torch-submit job logs <job_id>
Configuration
Torch Submit stores cluster configurations in ~/.cache/torch-submit/config.yaml
. You can manually edit this file if needed, but it's recommended to use the CLI commands for cluster management.
Requirements
- Python 3.7+
- PyTorch (for your actual jobs)
- SSH access to all machines in your cluster
Contributing
We welcome contributions! Please see our Contributing Guide for more details.
License
Torch Submit is released under the MIT License. See the LICENSE file for more details.
Support
If you encounter any issues or have questions, please file an issue on our GitHub Issues page.
Happy distributed training!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for torch_submit-0.1.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f8e5456ca0b21503b8518d3436bf5a3231247f3ca37f2f3c16f74b99d9fa8931 |
|
MD5 | d7786fe3a96ecdb596970e0e9dfc0ed6 |
|
BLAKE2b-256 | cfaef01be7d3d5d169ffb3420bb68f956e4996d3ae1853d07c3ba02d0834793d |