ods 0.0.3

Python client library for Staroid cloud platform

Open data studio python library

Install

pip install ods

Quick start

Initialize

1. Login staroid.com and get an access token. And set STAROID_ACCESS_TOKEN environment variable. See here for more detail.
2. Login staroid.com and create a SKE (Star Kubernetes engine) cluster.

import ods
# 'ske' is the name of kubernetes cluster created from staroid.com. Alternatively, you can export 'STAROID_SKE' environment variable.
ods.init(ske="kube-cluster-1")

Spark

Create spark session

Create spark session with default configuration

import ods
spark = ods.spark("spark-1") # 'spark-1' is name of spark-serverless instance to create.
df = spark.createDataFrame(....)

Configurue initial number of worker nodes

import ods
spark = ods.spark("spark-1", worker_num=3).session()
df = spark.createDataFrame(....)

detal=True to automatically download & configure delta lake

import ods
spark = ods.spark("spark-delta", delta=True)
spark.read.format("delta").load(....)

pass spark_conf dictionary for additonal configuration

import ods
spark = ods.spark(spark_conf = {
    "spark.hadoop.fs.s3a.access.key": "...",
    "spark.hadoop.fs.s3a.secret.key" : "..."
})

Check tests/test_spark.py as well.

Dask

Coming soon 🚛

import ods
cluster = ods.dask("dask-1", worker_num=10)

from dask.distributed import Client
client = Client(cluster)

Ray

Coming soon 🚛

import ods
ods.ray(cluster_name="")