datazone 1.2.0

Datazone Client Library 👋

Installation

You can install package via pip

pip install datazone

The CLI installs without any data processing libraries. The SDK dataset helpers need extras:

pip install "datazone[delta]"   # DatazoneClient.get_dataset_delta_table (deltalake)
pip install "datazone[pandas]"  # DatazoneClient.get_dataset_as_pandas (deltalake + pandas)

DatazoneDatabaseClient.query_df / query_np / query_arrow need pandas, numpy or pyarrow respectively.

Configuration & Login

Run the following command in your terminal to create a new profile.

datazone profile create

When prompted, enter the following information:

Profile Name: <PROFILE_NAME>
Host [app.datazone.co]: <DATAZONE_ADDRESS>
API Key: <API Key>

Then you can run following command to test it. (At the first profile creation, it will test automatically)

datazone auth test

Create first repository

datazone repository create hello-world
cd hello-world

It creates a new folder in your current directory. Thıs folder has two file initially. config.yaml and first_pipeline.py. After you modify your script you can deploy project via following command.

datazone repository deploy

After the deployment is complete, you can check your repository status via following command.

datazone repository summary

If repository has deployed, you can run your first execution with following command.

datazone execution run <pipeline_id>

Commands

Also you can check all command with help sub command. datazone auth --help or datazone repository summary --help

# Auth
datazone auth test

# Profile
datazone profile list
datazone profile create
datazone profile delete <profile_name>
datazone profile setdefault <profile_name>

# Repository
datazone repository list
datazone repository create
datazone repository deploy
datazone repository summary <file_name>
datazone repository clone <repository_id>
datazone repository pull

# Dataset
datazone dataset list
datazone dataset show <dataset_id> [--size <n>] [--transaction-id <id>] [--query "<SQL>"]
datazone dataset info|update|delete|schema|profile|refresh <dataset_id>
datazone dataset export <dataset_id> [--file-type <type>] [--query "<SQL>"] [--output <file>]
datazone dataset table <table_name>
datazone dataset branch show|rename <dataset_id> <branch_name>
datazone dataset transactions <dataset_id>

# Source
datazone source create
datazone source list
datazone source delete <source_id>
datazone source show <source_id>
datazone source test-connection [<source_id>] [--file <source.yml>]
datazone source explore|infer-schema

# Extract
datazone extract create
datazone extract list
datazone extract delete <extract_id>
datazone extract execute <extract_id>

# Schedule
datazone schedule create
datazone schedule list
datazone schedule delete <source_id>

# Execution
datazone execution run [--extract-id] [--pipeline-id] [<execution_type>] [<transform_selection>]
datazone execution list [--extract-id] [--pipeline-id]
datazone execution log <execution_id>
datazone execution list --dataset-id <dataset_id>
datazone execution show|summary|steps|transforms|outputs|upstreams|pod-metrics <execution_id>
datazone execution terminate <execution_id>
datazone execution rerun <execution_id> [--failed-only] [--follow]

# Pipeline
datazone pipeline create
datazone pipeline list
datazone pipeline delete <pipeline_id>
datazone pipeline show|lineage <pipeline_id>
datazone pipeline transform <transform_id>

# View
datazone view create
datazone view list [--dataset-id <dataset_id>]
datazone view delete <view_id>
datazone view show|update|refresh|profile <view_id>
datazone view validate

# SQL
datazone sql "<QUERY>" [--size <n>]

# Project
datazone project activities list <project_id>
datazone project show|update|delete <project_id>
datazone project overview|activity-logs|catalog|data-flow <project_id>
datazone project permissions <project_id> [--branch <branch>]

# Endpoint
datazone endpoint list|show|create|delete
datazone endpoint playground <endpoint_id>
datazone endpoint call <slug> [--filter key=value]
datazone endpoint validate

# Flow
datazone flow list [--branch <branch>] [--project-id <project_id>]
datazone flow show <flow_id> [--branch <branch>]
datazone flow create [--name <name>] [--project-id <project_id>] [--file <definition.yml>]
datazone flow update <flow_id> --file <definition.yml> [--branch <branch>]
datazone flow delete <flow_id>
datazone flow summaries <flow_id>...
datazone flow run <flow_id> [--parameters '<json>'] [--branch <branch>]
datazone flow validate --file <document.yml>
datazone flow node-types [<node_type>] [--schema]
datazone flow templates [<template_id>]
datazone flow runs list <flow_id>
datazone flow runs show <run_id>
datazone flow runs logs <run_id> [--node-id <node_id>]
datazone flow runs cancel <run_id>
datazone flow schedule list [--flow-id <flow_id>]
datazone flow schedule create <flow_id> --name <name> --expression "<cron>"
datazone flow schedule update <schedule_id> [--expression "<cron>"] [--active/--inactive]
datazone flow schedule delete <schedule_id>

# Knowledge Object
datazone knowledge-object list [--branch <branch>] [--project-id <project_id>]
datazone knowledge-object show <object_id>
datazone knowledge-object create [--project-id <project_id>] [--file <metadata.yml>]
datazone knowledge-object update <object_id> --file <metadata.yml>
datazone knowledge-object delete <object_id>
datazone knowledge-object graph <project_id>
datazone knowledge-object instance list <object_id> [--fields <a,b>] [--filters <expr>]
datazone knowledge-object instance get <object_id> <key> [--with-relationships]
datazone knowledge-object instance create <object_id> --data '<json>' | --file <file>
datazone knowledge-object instance batch <object_id> --file <instances.yml>
datazone knowledge-object instance update <object_id> <key> --data '<json>'
datazone knowledge-object instance delete <object_id> <key>
datazone knowledge-object instance history <object_id> <key>

# Query Explorer
datazone query execute "<QUERY>" --project-id <project_id> [--tab-id <tab_id>]
datazone query results [<result_id>] [--project-id <project_id>]
datazone query parse-tables "<QUERY>"
datazone query saved list|show|create|update|delete
datazone query tab list|show|create|update|delete

# Variable & Tag
datazone variable list|show|create|delete
datazone tag list|show|create|delete

# Notebook & Kernel
datazone notebook list|show|create|delete
datazone notebook kernel start <notebook_id> [--type <type>] [--compute <size>]
datazone notebook kernel show|stop|restart|interrupt <kernel_id>
datazone notebook kernel active <notebook_id>
datazone notebook kernel history <notebook_id>

# Alert, Action & Agent
datazone alert list|show|create|delete
datazone alert test --query "<QUERY>" --project-id <project_id>
datazone alert events [--alert-id <alert_id>]
datazone action list|show|delete
datazone action test <action_id> [--parameters '<json>'] [--timeout <seconds>]
datazone action calls [--action-id <action_id>]
datazone agent list|show|create|update|delete
datazone agent models [--kind CHAT|EMBEDDING] [--all]
datazone agent runtime-settings|embedding-token <agent_id> [--rotate]
datazone agent upload <agent_id> <file>
datazone agent chat-file <agent_id> <path> [--output <file>]
datazone agent logs <agent_id> [--question-id <question_id>]
datazone agent chat list|show|create|delete|questions|stop-question|feedback|visibility
datazone agent chat ask <chat_id> "<question>"
datazone agent chat playground <agent_id> [--reset]

# Vector
datazone vector list|show|delete
datazone vector search <vector_id> "<term>" [--limit <n>]
datazone vector reindex <vector_id>
datazone vector inspect

# Intelligent Apps
datazone app list [--branch <branch>]
datazone app show <app_id> [--definition]
datazone app embedding-token <app_id> [--rotate]
datazone app refresh-preview|chats <app_id>
datazone app from-yaml|to-yaml
datazone app chat list|show|create|questions
datazone app chat ask <chat_id> "<question>"
datazone app report list|show|delete
datazone app report trigger <report_id>
datazone app report records [--report-id <report_id>]
datazone app report record <record_id>

# Target, Model Account & Notification Channel
datazone target list|show|create|delete
datazone model-account list|show|create|update|delete
datazone model-account models|refresh-models|update-models <model_account_id>
datazone notification-channel list|show|create|delete

# Transaction & Favorite
datazone transaction list <dataset_id>
datazone transaction show <transaction_id>
datazone transaction daily-activity <dataset_id> [--days <n>]
datazone transaction file <transaction_id> <path>
datazone favorite list
datazone favorite add <entity_id> --type <ENTITY_TYPE>
datazone favorite remove <entity_id>

# Users, Roles & Policies
datazone user list|show|create|delete
datazone user me
datazone user invite <email> [--role-id <role_id>]...
datazone user bulk-invite [<email>...] [--file <emails.csv>] [--role-id <role_id>]
datazone user update [<user_id>]
datazone user change-password
datazone user permissions [--entity-id <id> --entity-type <type>]
datazone user recent
datazone user preferences show|update
datazone role list|show|create|delete
datazone role assign-user <user_id> <role_id>...        # replaces the user's roles
datazone role assign-policy <role_id> <policy_id>...    # replaces the role's policies
datazone role assign-role <role_id> <user_id>...
datazone role assigned-users|assigned-policies <role_id>
datazone role user-roles <user_id>
datazone policy list|show|create|update|delete
datazone policy assigned-roles <policy_id>

# Organisation & Keys
datazone organisation current
datazone organisation list
datazone organisation settings [<organisation_id>]
datazone organisation update-settings <general|auth|compute|domain|email|execution-notification|orion> --file <f>
datazone organisation update
datazone organisation check-domain-alias <alias>
datazone organisation usage [<organisation_id>] [--summary] [--detail --from-date <d> --to-date <d>] [--resource <r>]
datazone organisation account|invoices
datazone organisation audit-logs [--from-date <date>] [--to-date <date>] [--metrics]
datazone organisation orion-logs [--question-id <question_id>]
datazone organisation analytics show <compute|query|storage|token> [--metrics]
datazone organisation subscription-status
datazone organisation quota list|create|update|delete
datazone api-key list|create|delete
datazone access-key list|create|delete
datazone access-key validate <access_key_id> <secret_key_id>

# System
datazone system health
datazone system version
datazone system settings
datazone system domain-info
datazone system search [<term>] [--type <entity_type>] [--project-id <project_id>]

# Data Quality
datazone ruleset list [--project-id <project_id>] [--entity-id <entity_id>]
datazone ruleset show|delete|trigger <ruleset_id>
datazone ruleset create --file <ruleset.yml> --project-id <project_id> --entity-id <dataset_id>
datazone ruleset update <ruleset_id> [--file <ruleset.yml>] [--status <status>]
datazone ruleset validate|preview --file <ruleset.yml> --project-id <project_id> --entity-id <dataset_id>
datazone ruleset preview-sql|rule-history <ruleset_id> <rule_alias>
datazone ruleset runs|score-history [<ruleset_id>] [--project-id <project_id>]
datazone ruleset run <run_id>
datazone ruleset issues [--project-id <project_id>] [--ruleset-id <ruleset_id>]
datazone ruleset summary --dataset-id <dataset_id> | --project-id <project_id>
datazone ruleset template list|show|create|update|delete
datazone ruleset template generate-yaml <template_id>...

# Widget, Studio App & Collection
datazone widget list|show|create|update|delete
datazone studio-app list|show|create|update|delete
datazone studio-app build <studio_app_id> [--branch <branch>]
datazone studio-app builds list|show
datazone studio-app builds logs <build_id> [--follow]
datazone collection list|show|create|update|delete
datazone collection items <collection_id>
datazone collection add-item|remove-item <collection_id> <entity_id>
datazone collection builtin [<key>]

# Files & Uploads
datazone file-container list <project_id> [--prefix <dir/>]
datazone file-container download|metadata|delete <project_id> <path>
datazone file-container upload <project_id> <local_file> [--path <dir/>]
datazone file-container rename <project_id> <src> <dst>
datazone upload create-dataset <file>... --project-id <project_id> --name <name> [--preview]
datazone upload create-session|add-file|remove-file|preview|finish

# Common
datazone version
datazone info