A tool for running python code with runner on aws
Project description
Overview
The reality of ML training in universities is that we use what ever hardware we are given (for free). This means that we might have a few beefy GPU machines, an HPC cluster, plus some GCE/AWS credits that we get through grants. **Jaynes** is a well-designed python package that makes running across these inhomogenous hardward resources a pleasure.
install (requires unix operating system.)
pip install jaynes
To run locally:
import jaynes
def training(arg_1, key_arg=None):
print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')
jaynes.config(mode="local")
jaynes.run(training)
Setup
Jaynes has gone through various iterations. Now you can just copy-paste a jaynes.yml config file from our sample project to get started with running stuff across your horde of machines. When called from python, Jaynes automatically traverses the file tree to find the root of the project, the same way that git works.
For example, to run your code-block on a remote computer via ssh:
# your_project/jaynes.yml
version: 0
verbose: true
run: # this is specific to each launch, and is dynamically overwritten in-memory
mounts:
- !mounts.S3Code
s3_prefix: s3://ge-bair/jaynes-debug
local_path: .
host_path: /home/ubuntu/
container_path: /Users/geyang/learning-to-learn
pypath: true
excludes: "--exclude='*__pycache__' --exclude='*.git' --exclude='*.idea' --exclude='*.egg-info' --exclude='*.pkl'"
compress: true
runner:
!runners.Docker
name: # not implemented yet
image: "episodeyang/super-expert"
startup: "yes | pip install jaynes ml-logger -q"
work_directory: "{mounts[0].container_path}"
ipc: host
host:
envs: "LANG=utf-8"
pre_launch: "pip install jaynes ml-logger -q"
launch:
type: ssh
ip: <your ip address>
username: ubuntu
pem: ~/.ssh/your_rsa_key
In python (your code-block):
# your_project/launch.py
import jaynes
def training(arg_1, key_arg=None):
print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')
jaynes.run(training)
Using Modes
A lot of times you want to setup a different run modes so it is easy to switch between them during development.
# your_project/jaynes.yml
version: 0
mounts: # mount configurations Available keys: NOW, UUID,
- !mounts.S3Code &code-block_mount
s3_prefix: s3://ge-bair/jaynes-debug
local_path: .
host_path: /home/ubuntu/jaynes-mounts/{NOW:%Y-%m-%d}/{NOW:%H%M%S.%f}
# container_path: /Users/geyang/learning-to-learn
pypath: true
excludes: "--exclude='*__pycache__' --exclude='*.git' --exclude='*.idea' --exclude='*.egg-info' --exclude='*.pkl'"
compress: true
hosts:
hodor: &hodor
ip: <your ip address>
username: ubuntu
pem: ~/.ssh/incrementium-berkeley
runners:
- !runners.Docker &ssh_docker
name: "some-job" # only for docker
image: "episodeyang/super-expert"
startup: yes | pip install jaynes ml-logger -q
envs: "LANG=utf-8"
pypath: "{mounts[0].container_path}"
launch_directory: "{mounts[0].container_path}"
ipc: host
use_gpu: false
modes: # todo: add support to modes.
hodor:
mounts:
- *code-block_mount
runner: *ssh_docker
launch:
type: ssh
<<: *hodor
now run in python
# your_project/launch.py
import jaynes
def training(arg_1, key_arg=None):
print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')
jaynes.config(mode="hodor")
jaynes.run(training)
ToDos
[ ] more documentation
[ ] slurm singularity support
[ ] GCE support
[ ] support using non-s3 code-block repo.
Done
[x] get the initial template to work
Installation
pip install jaynes
Usage (Show me the Mo-NAY!! :moneybag::money_with_wings:)
Check out the test_projects folder for projects that you can run. I just recenlty implemented a hugely improved API that uses a static yaml configuration file, so the documentations are still comming.
To try to get things to work via ssh/slurm/aws, see test_projects.
To Develop
git clone https://github.com/episodeyang/jaynes.git
cd jaynes
make dev
To test, run
make test
This make dev command should build the wheel and install it in your current python environment. Take a look at the https://github.com/episodeyang/jaynes/blob/master/Makefile for details.
To publish, first update the version number, then do:
make publish
Acknowledgements
This code-block is inspired by @justinfu’s doodad, which is in turn built on top of Peter Chen’s script.
This code-block is written from scratch to allow a more permissible open-source license (BSD). Go bears :bear: !!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.