Skip to main content
Donate to the Python Software Foundation or Purchase a PyCharm License to Benefit the PSF! Donate Now

Lightweight interface to launching jobs in the cloud

Project description

ncluster

By Yaroslav Bulatov, Andrew Shaw, Ben Mann

Installation

Install pip, tmux, Python 3.6 (see below), then

pip install -r https://raw.githubusercontent.com/yaroslavvb/ncluster/master/requirements.txt
pip install -U ncluster
export AWS_ACCESS_KEY_ID=AKIAIBATdf343
export AWS_SECRET_ACCESS_KEY=z7yKEP/RhO3Olk343aiP
export AWS_DEFAULT_REGION=us-east-1

Python API

import ncluster
task = ncluster.make_task(instance_type='p2.xlarge')
task.upload('myscript.py')
task.run('python myscript.py > out')
task.download('out')

Command-line tools

ncluster
ncluster ls
ncluster hosts
ncluster ls
ncluster ls <substring>
ncluster ssh # connects to latest instance
ncluster ssh <substring>  # connects to latest instance containing <substring>
ncluster ssh \'<exact match>\'
ncluster mosh <substring> 
ncluster kill <substring>    # terminates matching instances
ncluster kill \'<exact match>\'
ncluster stop <substring>    # stops matching instances
ncluster start <substring>   # starts matching stopped instances
ncluster nano       # starts a tiny instance

ncluster ssh_    # like ssh but works on dumb terminals
ncluster ls     
ncluster cat <fn>
ncluster cmd "some command to run remotely on AWS"

ncluster efs   # gives EFS info such as the mount command

nsync -n gpubox
nsync -n gpubox -t transformer-xl

nsync -t {target directory} -n {substring}

nsync -n gpubox # syncs . to ~ on gpubox
nsync -t transformer-xl -n 4gpubox  # syncs . to ~/transformer-xl on 4gpubox


{substring} selects the most recently launched instances whose name contains the substring. Empty string is a valid substring. Skipping -t will sync to ~ on remote machine. Sync seems to be 1 way (from local -> remote)

Docs

  • Some out-of-date docs with more info docs

Extra

An example of installing pip/tmux/python 3.6 on MacOS

  1. Download Anaconda distribution following https://conda.io/docs/user-guide/install/index.html
  2. Install tmux through homebrew: https://brew.sh/, then brew install tmux

Then

conda create -n new python=3.6 -y
conda activate new

Extra Deps:

brew install fswatch

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
ncluster-0.1.41-py3-none-any.whl (70.1 kB) Copy SHA256 hash SHA256 Wheel py3
ncluster-0.1.41.tar.gz (44.1 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page