sagecreator 0.0.36

Package to provision and install architecture in AWS

SageCreator

SageCreator is a package meant to simplify cluster setup for Machine Learning in AWS.

It does all the heavy lifting to get cluster up and running in a matter of minutes on any of the AWS instance type(s).

It is using spot instances by default which can significantly reduce total cost of running the cluster.

If spot instances are not available or the specified spot instance price is too low, it falls back to on-demand instances.

After provision you can access Jupyter notebook that can run your code against provisioned server(s)

Installation

Install and update using pip:

$ pip install sagecreator

Python is required and it is highly recommended to install and run the package in virtualenv. Supported in Python >= 3.5.0.

Prerequisites

AWS Account

To provision the cluster you need an AWS Account and IAM user either in Administrators group or with custom policy:

• Access Key ID

• Secret Access Key

• Key pair (optional) - if not provided it will be generated and stored locally

Execution

After the installation, configure the tool by specifying configuration parameters:

$ sage configure
Access key id: <AWS Access Key ID>
Secret access key: <AWS Secret Access Key>
Company: <Name of your organization>
Owner: <Name of your team>
Private key file: <Absolute path to private key file> (optional, if not provided will be generated by the key pair)

Company, Owner, Service are required - those are used as tags for each instance in the cluster.

Private key file - if provided, given private key file will be used to provision the cluster.

---

Provision the cluster.

Provision step can take up to 20 minutes depending on network connection, cluster size, and instance type.

$ sage provision
Service: <Name of your service>
Instance type [t3.small]: <Instance type> (optional, if not provided default t3.small will be used)
Spot instance price [0.1]: <Spot instance price> (optional, if not provided default price of $0.1 will be used)
Cluster size [1]: <Cluster size> (optional, if not provided default cluster with 1 node will be provisioned)

Important:

The tool provides no guarantee that the instance(s) will be provisioned at specified Spot instance price.

If specified Spot instance price is lower than current AWS spot instance price then on-demand instance(s) will be provisioned.

Thus, it is up to the user to ensure that specified price is high enough for the request to be fulfilled.

The tool does provide guarantee that the cluster with specified configuration will be provisioned and thus it has to fall back to on-demand instances if it can’t provision spot instances.

---

Show path of the cluster configuration file.

Though not necessary it is possible to manually edit that file with customizations prior to running provision step.

$ sage pwd

---

Terminate cluster. This operation terminates all cluster nodes matching tags tuple of Company, Owner, Service.

$ sage terminate

Jupyter access

Once provision step is done and cluster is up you can access jupyter notebook on http://localhost:9000

We have provided a sample notebook to train a model on Fashion MNIST dataset using CNN in Keras

Under the hood

The logic that orchestrates the cluster and deploys the software is written in Ansible