A tool for launching and running commands on a cluster of EC2 instances
Simple library and CLI to manage and work with clusters of EC2 instances. Multi-purpose, but created to make distributed deep learning infrastructure easier.
ec2-cluster is designed for simple distributed tasks where Kubernetes is overkill or where fast cluster spin up/down is crucial. Example use cases are running distributed deep learning on an expensive cluster or running distributed load testing from many EC2 instances.
- Provide the minimal set of features to run distributed deep learning training jobs on EC2 instances.
- Provide libraries, not a framework or platform.
- Make cluster environments reproducible to allow for parallelization of experiments
- Make cluster launches fast
- Be resilient to EC2 capacity limitations
- Encourage ephemeral infrastructure design
- Focus on iterative, not disruptive, improvements on the common methodology of manually launching EC2 instances, ssh-ing to them, configuring environments by hand and running scripts
ec2-cluster can be consumed in two ways:
- A CLI for launching, describing and deleting clusters.
- A python library for scripting.
This library has three main components:L
- infra: creating cluster infrastructure
- orch: orchestrating simple runtime cluster configuration (e.g. generate a hostfile with runtime IPs)
- control: running commands on the cluster
CLI Quick Start
Library Quick Start
Release history Release notifications
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size ec2_cluster-0.2.0a4-py3-none-any.whl (19.3 kB)||File type Wheel||Python version py3||Upload date||Hashes View hashes|
|Filename, size ec2_cluster-0.2.0a4.tar.gz (18.1 kB)||File type Source||Python version None||Upload date||Hashes View hashes|
Hashes for ec2_cluster-0.2.0a4-py3-none-any.whl