A simple launcher for spinning up and managing Ray clusters for Daft.
Project description
Daft Launcher CLI Tool
daft-launcher
is a simple launcher for spinning up and managing Ray clusters for daft
.
Goal
Getting started with Daft in a local environment is easy. However, getting started with Daft in a cloud environment is substantially more difficult. So much more difficult, in fact, that users end up spending more time setting up their environment than actually playing with our query engine.
Daft Launcher aims to solve this problem by providing a simple CLI tool to remove all of this unnecessary heavy-lifting.
Capabilities
What Daft Launcher is capable of:
- Spinning up clusters.
- Listing all available clusters (as well as their statuses).
- Submitting jobs to a cluster.
- Connecting to the cluster (to view the Ray dashboard and submit jobs using the Ray protocol).
- Spinning down clusters.
- Creating configuration files.
- Running raw SQL statements using Daft's SQL API.
Currently supported cloud providers
- AWS
- GCP
- Azure
Usage
Pre-requisites
You'll need some python package manager installed.
We recommend using uv
for all things python.
AWS
If you're using AWS, you'll need:
- A valid AWS account with the necessary IAM role to spin up EC2 instances. This IAM role can either be created by you (assuming you have the appropriate permissions). Or this IAM role will need to be created by your administrator.
- The AWS CLI installed and configured on your machine.
- To login using the AWS CLI. For full instructions, please look here.
Installation
Using uv
:
# create project
mkdir my-project
cd my-project
# initialize project and setup virtual env
uv init
uv venv
source .venv/bin/activate
# install launcher
uv pip install daft-launcher
Example
All interactions with Daft Launcher are primarily communicated via a configuration file.
By default, Daft Launcher will look inside your $CWD
for a file named .daft.toml
.
You can override this behaviour by specifying a custom configuration file.
# create a new configuration file
# will create a file named `.daft.toml` in the current working directory
daft init-config
# or optionally, pass in a custom name
daft init-config my-custom-config.toml
# spin up a cluster
daft up
# or optionally, pass in a custom config file
daft up -c my-custom-config.toml
# list all the active clusters (can have multiple clusters running at the same time)
daft list
# submit a directory and a command to run on the cluster
daft submit --working-dir <...> -- command arg1 arg2 ...
# or optionally, pass in a custom config file
daft submit -c my-custom-config.toml --working-dir $WORKING_DIR -- command arg1 arg2 ...
# run a direct SQL query against the daft query engine running in the remote cluster
daft sql -- "SELECT * FROM my_table WHERE column = 'value'"
# or optionally, pass in a custom config file
daft sql -c my-custom-config.toml -- "SELECT * FROM my_table WHERE column = 'value'"
# spin down a cluster
daft down
# or optionally, pass in a custom name
daft down -c my-custom-config.toml
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for daft_launcher-0.3.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c31aa8a01bb4c123809cc134007e1849a3aba6d018946c024a9109dc487481d4 |
|
MD5 | e0a5b75bfef09120e4b938d50e2663bf |
|
BLAKE2b-256 | 98640c176d7608922409e95631e0dcd0b037bc9d8870b8a4ad934f1165c4e5e2 |