Skip to main content

A tool for running python code with docker on aws

Project description

## Todo

  • [x] get the initial template to work

Done

Installation

pip install jaynes

Usage (Show me the Mo-NAY!! :moneybag::money_with_wings:)

Take a look at the folder test_projects/! These project scripts are used for test and development, so they should work out-of-the-box (if you have the right box ahem).

To run Local Docker

import os
from jaynes import Jaynes
from main import LOG_DIR, train, abs_path

J = Jaynes(os.getcwd(), bucket="ge-bair", log=LOG_DIR + "/startup.log")
J.mount_s3(local="https://github.com/episodeyang/jaynes/blob/master/", pypath=True)
J.mount_s3(local=".https://github.com/episodeyang/jaynes/blob/master/.https://github.com/episodeyang/jaynes/blob/master/", pypath=True, file_mask="""https://github.com/episodeyang/jaynes/blob/master/__init__.py https://github.com/episodeyang/jaynes/blob/master/jaynes""")
J.mount_output(s3_dir=LOG_DIR, local=LOG_DIR, remote=LOG_DIR, docker=abs_path, sync_s3=False)
J.run_local()
J.setup_docker_run("python:3.6", docker_startup_scripts=("pip install cloudpickle",), use_gpu=False)
J.run_local_docker(train, a="hey", b=[0, 1, 2], log_dir=abs_path)

To run on remote server with ssh

from jaynes import Jaynes
from main import LOG_DIR, train, abs_path

J = Jaynes(remote_cwd='/home/ubuntu/', bucket="ge-bair", log=LOG_DIR + "/startup.log")
J.mount_s3(local="https://github.com/episodeyang/jaynes/blob/master/", pypath=True)
J.mount_s3(local=".https://github.com/episodeyang/jaynes/blob/master/.https://github.com/episodeyang/jaynes/blob/master/", pypath=True, file_mask="""https://github.com/episodeyang/jaynes/blob/master/__init__.py https://github.com/episodeyang/jaynes/blob/master/jaynes""")
J.mount_output(s3_dir=LOG_DIR, local=LOG_DIR, remote=LOG_DIR, docker=abs_path)
J.run_local(verbose=True)
J.setup_docker_run("thanard/matplotlib", docker_startup_scripts=("pip install cloudpickle",), use_gpu=True)
J.make_launch_script(train, a="hey", b=[0, 1, 2], log_dir=LOG_DIR, dry=True, verbose=True)

To Launch EC2 instance and Run on Startup

make sure the instance terminate after docker run!

from jaynes import Jaynes
from main import LOG_DIR, train, abs_path

J = Jaynes(remote_cwd='/home/ubuntu/', bucket="ge-bair", log=LOG_DIR + "/startup.log")
J.mount_s3(local="https://github.com/episodeyang/jaynes/blob/master/", pypath=True)
J.mount_s3(local=".https://github.com/episodeyang/jaynes/blob/master/.https://github.com/episodeyang/jaynes/blob/master/", pypath=True, file_mask="""https://github.com/episodeyang/jaynes/blob/master/__init__.py https://github.com/episodeyang/jaynes/blob/master/jaynes""")
J.mount_output(s3_dir=LOG_DIR, local=LOG_DIR, remote=LOG_DIR, docker=abs_path)
J.run_local(verbose=True)
J.setup_docker_run("thanard/matplotlib", docker_startup_scripts=("pip install cloudpickle",), use_gpu=True)
J.make_launch_script(train, a="hey", b=[0, 1, 2], log_dir=LOG_DIR, dry=True, verbose=True, terminate_after_finish=True)
J.launch_and_run(region="us-west-2", image_id="ami-bd4fd7c5", instance_type="t2.micro", key_name="ge-berkeley",
                 security_group="torch-gym-prebuilt", is_spot_instance=True, spot_price=0.004,
                 iam_instance_profile_arn="arn:aws:iam::055406702465:instance-profile/main", dry=False)

Jaynes does the following:

To Develop

git clone https://github.com/episodeyang/jaynes.git
cd jaynes
make dev

To test, run

make test

This make dev command should build the wheel and install it in your current python environment. Take a look at the https://github.com/episodeyang/jaynes/blob/master/Makefile for details.

To publish, first update the version number, then do:

make publish

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

jaynes-0.1.1-py3-none-any.whl (10.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page