gen2-connector 1.0.0b2

Ray gen2 node provider

gen2-connector provides Ray plugin to manage clusters over ibm gen2 infrastructure

1. Install Ray release 1.4.1 using pip install ray[default]==1.4.1

2. Install gen2-connector on your machine

git clone [e-mail]:project-codeflare/gen2-connector.git
cd gen2-connector
pip install .

3. Configure ibm vpc

   • Generate API KEY

   • Create/Update security group to have SSH, Redis and Ray Dashboard ports open: 22, 8265 and 6379

4. Create cluster config file

   • Use interactive vpc_config tool to generate cluster.yaml configuration file

   vpc_config --iam_api_key ${IAM_API_KEY} --format ray --filename cluster.yaml
   

   • Select security group from previous step when prompted

   • The output of the vpc_config is a cluster config yaml file, e.g

   =================================================
   Cluster config file: /tmp/tmpkf0dztfk.yaml
   =================================================
   

   • Alternatively, update cluster config manually based on defaults.yaml

5. Use generated file to bring ray cluster up, e.g

ray up /tmp/tmpkf0dztfk.yaml

• After finished, find cluster head node and worker nodes ips:

ray get-head-ip /tmp/tmpkf0dztfk.yaml
ray get-worker-ips /tmp/tmpkf0dztfk.yaml

• To get status of the cluster

ray status --address PUBLIC_HEAD_IP:6379

• Use browser to open ray dashboard on PUBLIC_HEAD_IP:8265. Alternatively use ray dashboard to forward ray cluster dashboard to your localhost.

• Submit example task ray submit /tmp/tmpkf0dztfk.yaml templates/example.py