Tractoray

The tool for running Ray clusters on Tracto.ai. Allows you to easily deploy and manage Ray clusters within Tracto.ai infrastructure.

Features

• Launch Ray clusters with configurable resources.
• Support native ray dashboard and ray client.
• Support Ray Debugger.
• Flexible Docker image configuration.

To see all the configuration options for Ray, please check the output of the tractoray start --help command.

Installation

Install tractoray with Ray CLI and all required dependencies:

pip install -U "tractoray[ray]"

Usage

Basic Commands

To use tractoray, you need to specify the working directory, for example your homedir //home/<login>/tractoray.

Start a cluster:

tractoray start --workdir //your/cypress/path --node-count 2

An output of the command contains an instruction to connect to the cluster.

Check the cluster status:

tractoray status --workdir //your/yt/path

Also JSON format is supported:

tractoray status --workdir //your/cypress/path --format json

Stop the cluster:

tractoray stop --workdir //your/cypress/path

For detailed information about task submission, log reading, and other operations, please check the ray status command output.

Using Custom Docker Images

You have two options for Docker images:

1. Use the default image as base (recommended):

   FROM cr.eu-north1.nebius.cloud/e00faee7vas5hpsh3s/tractoray/default:2025-06-12-16-59-42-9a2ce5611
   
   # Add your dependencies
   RUN pip install your-package
   

2. Build from scratch:

   • Install tractoray via pip
   • Use the same version as in your local environment and install all necessary dependencies for CUDA and Infiniband.

   FROM python:3.12
   
   RUN pip install tractoray==<your-local-version>
   # Add other dependencies
   

The default image includes all necessary dependencies and configurations to run Ray clusters for ML workloads. Using it as a base image is recommended to ensure compatibility.

Connect to the Ray Head Node terminal

Connect to the Ray head node terminal via the yt run-job-shell command or the web terminal in the UI. The tractoratay status command shows all necessary instructions and links.

Ray Debugging

Currently, only the legacy Ray debugger is supported.

To use ray debugger on Tracto.ai:

1. Start your Ray cluster with RAY_DEBUG=legacy environment variable and the --ray-debugger-external option: tractoray start --workdir //your/cypress/path --ray-head-params="--ray-debugger-external" --ray-worker-params="--ray-debugger-external" --env-var "RAY_DEBUG=legacy"
2. Connect to the Ray head node terminal using yt run-job-shell or web UI (instructions and necessary links can be found in the output of ray start and ray status).
3. Run ray debug in the terminal on the head node to start the Ray Debugger.

You can read more about debugging Ray tasks in the official Ray documentation.

Environment Variables

• YT_LOG_LEVEL: Set logging level

Limitations

• Some Ray CLI options, like ray status, may not work properly on a laptop due to Ray authentication constraints. Use the Ray dashboard and the Ray SDK, or run Ray CLI on the head node instead.
• Ray Serve is not supported.
• Observability features are disabled by default. You can enable and configure them in your custom image.
• Only legacy Ray Debugger is supported.