run_vast

A command-line tool. Lets you put bash commands in markdown files, and runs them in parallel on many vast.ai instances.

Uses a waiting/running/fail/succeed state machine to represent every command. All state is contained in the markdown file, in human-readable and human-editable form.

I like to provision 10-20 Vast instances, usually with 4x4090s each, at the beginning of the day, with the same custom Dockerfile.

Vast lets me keep these nodes idle for very cheap. So by default, all instances are idle.

Then later, I will add a new ML experiment to my journal.md file. Every training run in the experiment is a bash command in a triple-backtick ```vast code block.

Then I run rv journal.md to run them all in parallel. Each Vast instance will go idle when its command succeeds.

If a run fails, the code block will be marked as ```vast:fail/012345, where 012345 is the instance ID of the machine it ran on. I can then ssh into the instance and debug my training run.

If a run starts up successfully, the code block will be marked as ```vast:running/012345.

Installation

pip install run_vast
rv journal.md

Usage

Make a list of commands you want to run

You should put these in a markdown file. Each command gets its own triple-backtick code block, annotated with vast.

For example, to train nanogpt with two different lrs:

# Train nanogpt with different lrs

lr=0.5 and lr=1.5:

```vast
git clone https://github.com/karpathy/nanogpt && \
cd nanogpt && \
pip install torch numpy transformers datasets tiktoken wandb tqdm && \
python data/shakespeare_char/prepare.py &&
python train.py config/train_shakespeare_char.py --min_lr=0.5e-4
```

```vast
git clone https://github.com/karpathy/nanogpt && \
cd nanogpt && \
pip install torch numpy transformers datasets tiktoken wandb tqdm && \
python data/shakespeare_char/prepare.py &&
python train.py config/train_shakespeare_char.py --min_lr=1.5e-4
```

Set up your Vast account

You need to make an SSH key to connect to Vast instances.

Register your SSH key on the vast website, then put the private key in ~/.ssh/id_vast.

Run rv my_training_runs.md

rv will prompt you to provision two Vast instances, so it can run both commands in parallel.

Important: in the vast.ai web UI, before provisioning Vast instances, you must edit the instance template to set the environment variable IS_FOR_AUTORUNNING=1.

Remember to press the "+" button to save the environment variable.

Go to the Vast dashboard and wait for your instances to be "Connected"

This should take a minute or so.

Then, return to the rv prompt and press Enter to continue.

Wait for your commands to finish

You should track your runs via i.e. wandb. rv doesn't handle any logging for you.

Once your commands have finished, run rv journal.md.

It will move them from the vast:running/0123456 state to the vast:finished state.

License

MIT License