bodosdk 1.0.0

Bodo Platform SDK

Bodo Platform SDK

A simple SDK for Bodo Cloud Platform.

List of contents:

• Getting Started
• Job resource
• Cluster resource
• Workspace resource
• Cloud config
• Instance Role Manager

Getting started

You will obtain client_id and secret_key. Those are needed for BodoClient definition.

from bodosdk.models import WorkspaceKeys
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)

You can also export client_id and secret_key as BODO_CLIENT_ID and BODO_SECRET_KEY, then you don't need to define keys.

from bodosdk.client import get_bodo_client

client = get_bodo_client()

other bodo client options

• print_logs - default False, if enabled all API calls will be printed

from bodosdk.client import get_bodo_client
from bodosdk.models import WorkspaceKeys

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys, print_logs=True)

Job resource

Module responsible for managing jobs in workspace.

• create job
• list jobs
• get job
• remove job
• get execution info
• waiter

Create job

BodoClient.job.create(job: JobDefinition)

Creates a job to be executed on cluster. You can either create job dedicated cluster by providing it's definition or provide existing cluster uuid. Job dedicated clusters will be removed as soon as job execution will finish, if you provide uuid of existing one, cluster will remain.

Example 1. Use git repository and cluster definition:

from bodosdk.models import GitRepoSource, WorkspaceKeys, JobDefinition, JobClusterDefinition
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)

job_definition = JobDefinition(
    name='test',
    args='./examples/nyc-taxi/get_daily_pickups.py',
    source_config=GitRepoSource(
        repo_url='https://github.com/Bodo-inc/Bodo-examples.git',
        username='XYZ',
        token='XYZ'
    ),
    cluster_object=JobClusterDefinition(
        instance_type='c5.large',
        accelerated_networking=False,
        image_id='ami-0a2005b824a8758e5',
        workers_quantity=2
    ),
    variables=[],
    timeout=120,
    retries=0,
    retries_delay=0,
    retry_on_timeout=False
)

client.job.create(job_definition)

Example 2. Run job from shared drive and existing cluster:

from bodosdk.models import JobCluster, WorkspaceSource, WorkspaceKeys, JobDefinition
from bodosdk.client import get_bodo_client
from uuid import UUID

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)

job_definition = JobDefinition(
    name='test',
    args='nyc-taxi/get_daily_pickups.py',
    source_config=WorkspaceSource(
        path='/shared/bodo-examples/examples/'
    ),
    cluster_object=JobCluster(
        uuid=UUID('0f0c5261-9827-4572-84f3-f6a9b10cf77d')
    ),
    variables=[],
    timeout=120,
    retries=0,
    retries_delay=0,
    retry_on_timeout=False
)

client.job.create(job_definition)

List jobs

BodoClient.job.list()

Returns list of all jobs defined in workspace.

Example:

from typing import List
from bodosdk.models import WorkspaceKeys, JobResponse
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
jobs: List[JobResponse] = client.job.list()

Get job

BodoClient.job.get(job_uuid)

Returns specific job in workspace. Example:

from bodosdk.models import WorkspaceKeys, JobResponse
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
job: JobResponse = client.job.get('8c32aec5-7181-45cc-9e17-8aff35fd269e')

Remove job

BodoClient.job.delete(job_uuid)

Removes specific job from workspace. Example:

from bodosdk.models import WorkspaceKeys
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
client.job.remove('8c32aec5-7181-45cc-9e17-8aff35fd269e')

Get execution

BodoClient.job.get_job_executions(job_uuid)

Gets all executions info for specific job. Result it's a list with one element (in future we might extend it)

from bodosdk.models import WorkspaceKeys, JobExecution
from bodosdk.client import get_bodo_client
from typing import List

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
executions: List[JobExecution] = client.job.get_job_executions('8c32aec5-7181-45cc-9e17-8aff35fd269e')

Job waiter

BodoClient.job.get_waiter()

Get waiter object, which can be used to wait till job finish. Waiter has following method

from typing import Callable
def wait(
        self,
        uuid,
        on_success: Callable = None,
        on_failure: Callable = None,
        on_timeout: Callable = None,
        check_period=10,
        timeout=None
):
  pass

By default returns job model if no callbacks is provided. There is option to pass callable objects as following parameters:

• on_success - will be executed on succes, job object passed as argument
• on_failure - will be executed on failure, job object passed as argument
• on_timeout - will be executed on timeout, job_uuid passed as argument

Other options are:

• check_period - seconds between status checks
• timeout - threshold in seconds after which Timeout error will be raised, None means no timeout

Example 1. Success callback:

from bodosdk.models import WorkspaceKeys
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
waiter = client.job.get_waiter()


def success_callback(job):
    print('Job has finished')
    return job


result = waiter.wait('8c32aec5-7181-45cc-9e17-8aff35fd269e', on_success=success_callback)

Example 2. Timeout callback:

from bodosdk.models import WorkspaceKeys
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
waiter = client.job.get_waiter()


def timeout_callback(job_uuid):
    print(f'Waiter timeout for {job_uuid}')
    return job_uuid


result = waiter.wait('8c32aec5-7181-45cc-9e17-8aff35fd269e', on_timeout=timeout_callback, timeout=1)

Cluster resource

Module responsible for managing clusters in workspace.

• get available instance types
• get available images
• create cluster
• list clusters
• get cluster
• remove cluster
• scale cluster

Available instance types

BodoClient.cluster.get_available_instance_types(region:str)

Returns list of instance types available for given region

from bodosdk.models import WorkspaceKeys
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
instance_types = client.cluster.get_available_instance_types('us-west-2')

Available images

BodoClient.cluster.get_available_images(region:str)

Returns list of images available for given region

from bodosdk.models import WorkspaceKeys
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
images = client.cluster.get_available_images('us-west-2')

Create cluster

BodoClient.cluster.create(cluster_definition: ClusterDefinition)

Creates cluster in workspace.

from bodosdk.models import WorkspaceKeys, ClusterDefinition
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
cluster_definition = ClusterDefinition(
    name="test",
    instance_type="c5.large",
    workers_quantity=2,
    auto_shutdown=100,
    auto_pause=100,
    image_id="ami-038d89f8d9470c862",
    bodo_version="2022.4",
    description="my desc here"
)
result_create = client.cluster.create(cluster_definition)

List clusters

BodoClient.cluster.list()

Returns list of all clusters in workspace

from bodosdk.models import WorkspaceKeys, ClusterResponse
from bodosdk.client import get_bodo_client
from typing import List

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
clusters: List[ClusterResponse] = client.cluster.list()

Get cluster

BodoClient.cluster.get(cluster_uuid)

Returns cluser by uuid

from bodosdk.models import WorkspaceKeys, ClusterResponse
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
clusters: ClusterResponse = client.cluster.get('<CLUSTER-UUID>')

Remove cluster

BodoClient.client.remove(cluster_uuid, force_remove=False, mark_as_terminated=False)

Method removing cluster from platform

• force_remove: try to remove cluster even if something on cluster is happeing
• mark_as_terminated: mark cluster as removed without removing resources, may be useful if cluster creation failed and common removing is failing

from bodosdk.models import WorkspaceKeys
from bodosdk.client import get_bodo_client
from typing import List

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
client.cluster.remove('<CLUSTER-UUID>')

Scale cluster

BodoClient.cluster.scale(scale_cluster: ScaleCluster)

Changes number of nodes in cluster (AWS only)

from bodosdk.models import WorkspaceKeys, ScaleCluster, ClusterResponse
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
NEW_WORKERS_QUANTITY = 3
scale_cluster = ScaleCluster(
    uuid='<CLUSTER-UUID>',
    workers_quantity=NEW_WORKERS_QUANTITY
)
cluster: ClusterResponse = client.cluster.scale(scale_cluster)

Workspace resource

Module responsible for managing workspaces in an organization.

• workspace getting started
• create workspace
• list workspaces
• get workspace
• remove workspace
• assign user

Workspace getting started

In order to work with Workspace, users need to generate Personal Tokens, under Admin Console, from the Bodo Platform Dashboard. Then instantiate a PersonalKeys object with the generated client_id and secret_id. Then Pass in this personal key while instantiating a client object

from bodosdk.models import PersonalKeys
personal_keys = PersonalKeys(
    client_id='<CLIENT-ID>',
    secret_id='<SECRET-ID>',
)
client = get_bodo_organization_client(personal_keys)

Create Workspace

BodoClient.workspace.create(workspace_definition: WorkspaceDefinition) Creates a workspace with the specifications passed in through a WorkspaceDefinition object under the user's organization

from bodosdk.models import PersonalKeys
from bodosdk.models import WorkspaceDefinition
personal_keys = PersonalKeys(
    client_id='<CLIENT-ID>',
    secret_id='<SECRET-ID>',
)
client = get_bodo_organization_client(personal_keys)
wd = WorkspaceDefinition(
    name="<WORSPACE-NAME>",
    cloud_config_uuid="<CONFIG-UUID>",
    region="<WORKSPACE-REGION>"
)
resp = client.workspace.create(wd)

List Workspaces

BodoClient.workspace.list() Returns a list of all workspaces defined under this organization. The with_task boolean controls printing out tasks running in the workspaces. The returned list is a list of GetWorkspaceResponse object

from bodosdk.models import PersonalKeys
personal_keys = PersonalKeys(
    client_id='<CLIENT-ID>',
    secret_id='<SECRET-ID>',
)
client = get_bodo_organization_client(personal_keys)
resp = client.workspace.list(with_tasks=False)

Get Workspace

BodoClient.workspace.get(uuid: str) Returns information about the workspace with the given uuid. Returns a GetWorkspaceResponse object with details about the workspace uuid mentioned.

from bodosdk.models import PersonalKeys
personal_keys = PersonalKeys(
    client_id='<CLIENT-ID>',
    secret_id='<SECRET-ID>',
)
client = get_bodo_organization_client(personal_keys)
resp = client.workspace.get("<WORKSPACE-UUID>")

Remove Workspace

BodoClient.workspace.remove(uuid: str) Removes the workspace with the passed in uuid. The operation is only successful if all resources within the workspaces(jobs, clusters, notebooks) are terminated. Otherwise, returns an error. Returns None if successful

from bodosdk.models import PersonalKeys
personal_keys = PersonalKeys(
    client_id='<CLIENT-ID>',
    secret_id='<SECRET-ID>',
)
client = get_bodo_organization_client(personal_keys)
resp = client.workspace.remove("<WORKSPACE-UUID>")

Assign user

BodoClient.workspace.remove(uuid: str) Assign user to workspace.

from bodosdk.models import PersonalKeys
personal_keys = PersonalKeys(
    client_id='<CLIENT-ID>',
    secret_id='<SECRET-ID>',
)
client = get_bodo_organization_client(personal_keys)
workspace_uuid = "<some uuid>"
users: List[UserAssignment] = [
    UserAssignment(
        email="[e-mail]",
        skip_email=True,
        bodo_role=BodoRole.ADMIN
    )
]
client.workspace.assign_users(workspace_uuid, users):

Cloud Config

Module responsible for creating cloud configurations for organization.

• create cloud configuration
• list cloud configurations
• get cloud config

Create config

BodoClient.cloud_config.create(config: Union[CreateAwsCloudConfig, CreateAzureCloudConfig])

Create cloud configuration for cloud

AWS example

from bodosdk.models import OrganizationKeys, CreateAwsProviderData, CreateAwsCloudConfig, AwsCloudConfig
from bodosdk.client import get_bodo_client

keys = OrganizationKeys(
    client_id='XYZ',
    secret_key='XYZ'
)

client = get_bodo_client(keys)

config = CreateAwsCloudConfig(
    name='test',
    aws_provider_data=CreateAwsProviderData(
        tf_backend_region='us-west-1',
        access_key_id='xyz',
        secret_access_key='xyz'
    )

)
config: AwsCloudConfig = client.cloud_config.create(config)

Azure example

from bodosdk.models import OrganizationKeys, CreateAzureProviderData, CreateAzureCloudConfig, AzureCloudConfig
from bodosdk.client import get_bodo_client

keys = OrganizationKeys(
    client_id='XYZ',
    secret_key='XYZ'
)

client = get_bodo_client(keys)

config = CreateAzureCloudConfig(
    name='test',
    azure_provider_data=CreateAzureProviderData(
        tf_backend_region='eastus',
        tenant_id='xyz',
        subscription_id='xyz',
        resource_group='MyResourceGroup'
    )
    
)
config: AzureCloudConfig = client.cloud_config.create(config)

Get config

BodoClient.cloud_config.list()

Get list of cloud configs.

from bodosdk.models import OrganizationKeys, AzureCloudConfig, AwsCloudConfig
from bodosdk.client import get_bodo_client
from typing import Union, List

keys = OrganizationKeys(
    client_id='XYZ',
    secret_key='XYZ'
)

client = get_bodo_client(keys)

configs: List[Union[AwsCloudConfig, AzureCloudConfig]] = client.cloud_config.list()

Get config

BodoClient.cloud_config.get(uuid: Union[str, UUID])

Get cloud config by uuid.

from bodosdk.models import OrganizationKeys, AzureCloudConfig, AwsCloudConfig
from bodosdk.client import get_bodo_client
from typing import Union

keys = OrganizationKeys(
    client_id='XYZ',
    secret_key='XYZ'
)

client = get_bodo_client(keys)

config: Union[AwsCloudConfig, AzureCloudConfig] = client.cloud_config.get('8c32aec5-7181-45cc-9e17-8aff35fd269e')

Instance Role Manager

Module responsible for managing AWS roles in workspace.

• create role
• list roles
• get role
• remove role

Create role

BodoClient.instance_role.create()

Creates an AWS role with the specified role definition with a given AWS role arn.

from bodosdk.models import WorkspaceKeys, CreateRoleDefinition, CreateRoleResponse
from bodosdk.client import get_bodo_client
from typing import List

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
role_definition = CreateRoleDefinition(
    name="test-sdk-role-creation",
    description="testing",
    data=InstanceRole(role_arn="arn:aws:iam::427443013497:role/testing")
)
result_create:CreateRoleResponse = client.instance_role.create(role_definition)

List roles

BodoClient.instance_role.list()

Returns list of all roles in workspace

from bodosdk.models import WorkspaceKeys, InstanceRoleItem
from bodosdk.client import get_bodo_client
from typing import List

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
result_list:List[InstanceRoleItem] = client.instance_role.list()

Get role

BodoClient.instance_role.get(cluster_uuid)

Returns role by uuid

from bodosdk.models import WorkspaceKeys, InstanceRoleItem
from bodosdk.client import get_bodo_client

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
clusters: InstanceRoleItem = client.instance_role.get('<CLUSTER-UUID>')

Remove role

BodoClient.instance_role.remove(cluster_uuid, mark_as_terminated=False)

Method removing role from a workspace

• mark_as_terminated: mark role as removed without removing resources, may be useful if role creation failed and common removing is failing

from bodosdk.models import WorkspaceKeys
from bodosdk.client import get_bodo_client
from typing import List

keys = WorkspaceKeys(
    client_id='XYZ',
    secret_key='XYZ'
)
client = get_bodo_client(keys)
client.instance_role.remove('<ROLE-UUID>')