Lightweight interface to launching jobs in the cloud
Project description
ncluster
By Yaroslav Bulatov, Andrew Shaw, Ben Mann
https://github.com/cybertronai/ncluster
Installation
Install pip, tmux, Python 3.6 (see below), then
pip install -r https://raw.githubusercontent.com/yaroslavvb/ncluster/master/requirements.txt
pip install -U ncluster
export AWS_ACCESS_KEY_ID=AKIAIBATdf343
export AWS_SECRET_ACCESS_KEY=z7yKEP/RhO3Olk343aiP
export AWS_DEFAULT_REGION=us-east-1
Python API
import ncluster
task = ncluster.make_task(instance_type='p2.xlarge')
task.upload('myscript.py')
task.run('python myscript.py > out')
task.download('out')
Command-line tools
ncluster
ncluster ls
ncluster hosts
ncluster ls
ncluster ls <substring>
ncluster ssh # connects to latest instance
ncluster ssh <substring> # connects to latest instance containing <substring>
ncluster ssh \'<exact match>\'
ncluster mosh <substring>
ncluster kill <substring> # terminates matching instances
ncluster kill \'<exact match>\'
ncluster stop <substring> # stops matching instances
ncluster start <substring> # starts matching stopped instances
ncluster nano # starts a tiny instance
ncluster ssh_ # like ssh but works on dumb terminals
ncluster ls
ncluster cat <fn>
ncluster cmd "some command to run remotely on AWS"
ncluster efs # gives EFS info such as the mount command
nsync -n gpubox
nsync -n gpubox -t transformer-xl
nsync -t {target directory} -n {substring}
nsync -n gpubox # syncs . to ~ on gpubox
nsync -t transformer-xl -n 4gpubox # syncs . to ~/transformer-xl on 4gpubox
{substring} selects the most recently launched instances whose name contains the substring. Empty string is a valid substring. Skipping -t will sync to ~ on remote machine. Sync seems to be 1 way (from local -> remote)
Docs
- Some out-of-date docs with more info docs
Extra
An example of installing pip/tmux/python 3.6 on MacOS
- Download Anaconda distribution following https://conda.io/docs/user-guide/install/index.html
- Install tmux through homebrew: https://brew.sh/, then
brew install tmux
Then
conda create -n new python=3.6 -y
conda activate new
Extra Deps:
brew install fswatch
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
ncluster-0.1.46.tar.gz
(46.7 kB
view hashes)
Built Distribution
ncluster-0.1.46-py3-none-any.whl
(73.1 kB
view hashes)
Close
Hashes for ncluster-0.1.46-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6265be1f31667ab5123ee2543ff3f35bcad18ebcd65d9f494454363e496bd3c5 |
|
MD5 | 5cab5b8650d832e1eb3b115713dd3102 |
|
BLAKE2b-256 | 72631f7a5d01393048e63d44e72bbfbb0ddb45c70adc27a1d3d94d76d4a36abc |