aoa 6.1.1

Python client for Teradata AnalyticOps Accelerator (AOA)

Teradata AnalyticOps Client

• Official Documentation
• Release Notes

CLI

Installation

The cli is available from pypi here

pip install aoa

Overview of CLI

The cli can be used to perform a number of interactions and guides the user to perform those actions.

> aoa -h
usage: aoa [-h] [--debug] [--version] {list,add,retire,run,init,clone,link,message,connection} ...

AOA CLI

optional arguments:
  -h, --help            show this help message and exit
  --debug               Enable debug logging
  --version             Display the version of this tool

actions:
  valid actions

  {list,add,retire,run,init,clone,link,message,connection}
    list                List projects, models, local models or datasets
    add                 Add model to working dir
    retire              Retire active deployments for all users/projects
    run                 Train and Evaluate model locally
    init                Initialize model directory with basic structure
    clone               Clone Project Repository
    link                Link repo to Project Repository
    message             Send a message to AOA message broker
    connection          Manage local connections

aoa clone

The clone command provides a convenient way to perform a git clone of the repository associated with a given project. The command can be run interactively and will allow you to select the project you wish to clone. Note that by default it clones to the current working directory so you either need to make sure you create an empty folder and run it from within there or else provide the --path argument.

> aoa clone -h
usage: aoa clone [-h] [--debug] [-id PROJECT_ID] [-p PATH]

optional arguments:
  -h, --help            show this help message and exit
  --debug               Enable debug logging
  -id PROJECT_ID, --project-id PROJECT_ID
                        Id of Project to clone
  -p PATH, --path PATH  Path to clone repository to

aoa init

When you create a git repository, its empty by default. The init command allows you to initialize the repository with the structure required by the AOA. It also adds a default README.md and HOWTO.md.

> aoa init -h
usage: aoa init [-h] [--debug]

optional arguments:
  -h, --help  show this help message and exit
  --debug     Enable debug logging

aoa list

Allows to list the aoa resources. In the cases of listing models (pushed / committed) and datasets, it will prompt the user to select a project prior showing the results. In the case of local models, it lists both committed and non-committed models.

> aoa list -h
usage: aoa list [-h] [--debug] [-p] [-m] [-lm] [-t] [-d] [-c]

optional arguments:
  -h, --help           show this help message and exit
  --debug              Enable debug logging
  -p, --projects       List projects
  -m, --models         List registered models (committed / pushed)
  -lm, --local-models  List local models. Includes registered and non-
                       registered (non-committed / non-pushed)
  -t, --templates      List dataset templates
  -d, --datasets       List datasets
  -c, --connections    List local connections

All results are shown in the format

[index] (id of the resource) name

for example:

List of models for project Demo:
--------------------------------
[0] (03c9a01f-bd46-4e7c-9a60-4282039094e6) Diabetes Prediction
[1] (74eca506-e967-48f1-92ad-fb217b07e181) IMDB Sentiment Analysis

aoa add

Adding a new model to a given repository requires a number of steps. You need to create the folder structure, configuration files, generate a modelId, etc. The add command is intended to simplify this for the user. It will interactively prompt you for the model name, language, description and even allow you to use a model template to get you started. This can really help reduce the boilerplate required and ensure you get started developing quicker while maintaining a standard repository structure.

> aoa add
model name: my new model
model description: to show adding new models
These languages are supported: R, python, sql
model language: python
templates available for python: empty, sklearn
template type (leave blank for the default one): 

aoa run

The cli can be used to validate the model training and evaluation logic locally before committing to git. This simplifies the development lifecycle and allows you to test and validate many options. It also enables you to avoid creating the dataset definitions in the AOA UI until you are ready and have a finalised version.

> aoa run -h
usage: aoa run [-h] [--debug] [-id MODEL_ID] [-m MODE] [-d DATASET_ID]
                         [-t DATASET_TEMPLATE_ID] [-ld LOCAL_DATASET] [-lt LOCAL_DATASET_TEMPLATE]
                         [-c CONNECTION]

optional arguments:
  -h, --help            show this help message and exit
  --debug               Enable debug logging
  -id MODEL_ID, --model-id MODEL_ID
                        Id of model
  -m MODE, --mode MODE  Mode (train or evaluate)
  -d DATASET_ID, --dataset-id DATASET_ID
                        Remote datasetId
  -t DATASET_TEMPLATE_ID, --dataset-template-id DATASET_TEMPLATE_ID
                        Remote datasetTemplateId
  -c CONNECTION, --connection CONNECTION
                        Local connection id

You can run all of this as a single command or interactively by selecting some optional arguments, or none of them.

For example, if you want to run the cli interactively you just select aoa run but if you wanted to run it non interactively to train a given model with a given datasetId you would expect

> aoa run -id <modelId> -m <mode> -d <datasetId>

And if you wanted to select the model interactively but use a specific local dataset definition, you would execute

> aoa run -ld /path/to/my_test_dataset.json

aoa connection

As for security reasons, the connection credentials stored in the core service can not be accessed locally through the client. In order to solve that, users may list, create and remove local connection credentials to be used with the run command.

> aoa connection -h
usage: aoa connection [-h] {list,add,remove,export} ...

optional arguments:
  -h, --help         show this help message and exit

actions:
  valid actions

  {list,add,remove}
    list             List all local connections
    add              Add a local connection
    remove           Remove a local connection
    export           Export a local connection to be used as a shell script

aoa connection list

List all local connections.

> aoa connection list -h
usage: aoa connection list [-h] [--debug]

optional arguments:
  -h, --help  show this help message and exit
  --debug     Enable debug logging

aoa connection add

Add a local connection.

> aoa connection add -h
usage: aoa connection add [-h] [--debug] [-n NAME] [-H HOST] [-u USERNAME] [-p PASSWORD]

optional arguments:
  -h, --help            show this help message and exit
  --debug               Enable debug logging
  -n NAME, --name NAME  Connection name
  -H HOST, --host HOST  Connection host
  -u USERNAME, --username USERNAME
                        Connection username
  -p PASSWORD, --password PASSWORD
                        Connection password

aoa connection remove

Remove a local connection.

> aoa connection remove -h
usage: aoa connection remove [-h] [--debug] [-c CONNECTION]

optional arguments:
  -h, --help            show this help message and exit
  --debug               Enable debug logging
  -c CONNECTION, --connection CONNECTION
                        Local connection id

aoa connection export

Export a local connection to be used as a shell script.

> aoa connection export -h
usage: aoa connection export [-h] [--debug] [-c CONNECTION]

optional arguments:
  -h, --help            show this help message and exit
  --debug               Enable debug logging
  -c CONNECTION, --connection CONNECTION
                        Local connection id

Python SDK

Python client for Teradata AnalyticOps allows users to integrate with AnalyticOps from anywhere they can execute python such as notebooks, IDEs etc. It can also be used for devops to automate additional parts of the process and integreate into the wider organization.

• Installation
• Create Client
• Read Entities
• Deploy Model Version
• Import Model Version (BYOM)

Installation

The cli is available from pypi here

pip install aoa

Create Client

By default, creating an instance of the AoaClient() will use the users aoa configuration stored in ~/.aoa/config.yaml. You can override these values by passing the relevant constructor arguments or even with env variables.

:linenos:

 from aoa import AoaClient

 client = AoaClient(project_id="23e1df4b-b630-47a1-ab80-7ad5385fcd8d")

Read Entities

We provide an extensive sdk implementation to interact with the APIs. You can find, create, update, archive, etc any entity that supports it via the SDK. In addition, most if not all search endpoints are also implemented in the sdk. Here are some examples

:linenos:

 from aoa import AoaClient, DatasetApi, JobApi
 import pprint

 client = AoaClient(project_id="23e1df4b-b630-47a1-ab80-7ad5385fcd8d")

 dataset_api = DatasetApi(aoa_client=client)

 datasets = dataset_api.find_all()
 pprint.pprint(datasets)

 dataset = dataset_api.find_by_id("11e1df4b-b630-47a1-ab80-7ad5385fcd8c")
 pprint.pprint(dataset)

 job_api = JobApi(aoa_client=client)
 jobs = job_api.find_by_id("21e1df4b-b630-47a1-ab80-7ad5385fcd1c")
 pprint.pprint(jobs)

Deploy Model Version

Let's assume we have a model version 4131df4b-b630-47a1-ab80-7ad5385fcd15 which we want to deploy In-Vantage and schedule it to execute once a month at midnight of the first day of the month using dataset connection 11e1df4b-b630-47a1-ab80-7ad5385fcd8c and dataset template d8a35d98-21ce-47d0-b9f2-00d355777de1. We can use the SDK as follows to perform this.

:linenos:

 from aoa import AoaClient, TrainedModelApi, JobApi

 client = AoaClient(project_id="23e1df4b-b630-47a1-ab80-7ad5385fcd8d")
 trained_model_api = TrainedModelApi(aoa_client=client)
 job_api = JobApi(aoa_client=client)

 trained_model_id = "4131df4b-b630-47a1-ab80-7ad5385fcd15"
 deploy_request = {
   "type": "IN_VANTAGE",
   "datasetConnectionId": "11e1df4b-b630-47a1-ab80-7ad5385fcd8c",
   "datasetTemplateId": "d8a35d98-21ce-47d0-b9f2-00d355777de1",
   "args": {
      "CRON": "0 0 1 * *"
   }
 }

 job = trained_model_api.deploy(trained_model_id, deploy_request)

 # wait until the job completes (if the job fails it will raise an exception)
 job_api.wait(job['id'])

Import Model Version

Let's assume we have a PMML model which we have trained in another data science platfrom. We want to import the artefacts for this version (model.pmml and data_stats.json) against a BYOM model f937b5d8-02c6-5150-80c7-1e4ff07fea31.

:linenos:

from aoa import AoaClient, TrainedModelApi, TrainedModelArtefactsApi, JobApi
import uuid

client = AoaClient(project_id="23e1df4b-b630-47a1-ab80-7ad5385fcd8d")
trained_model_api = TrainedModelApi(aoa_client=client)
trained_model_artefacts_api = TrainedModelArtefactsApi(aoa_client=client)
job_api = JobApi(aoa_client=client)

artefacts_import_id = uuid.uuid4()
artefacts = ["model.pmml", "data_stats.json"]

# first, upload the artefacts which we want to associate with the BYOM model version
trained_model_artefacts_api.upload_artefacts(artefacts_import_id, artefacts)

import_request = {
    # tell the import job how to find the already uploaded artefacts so it can link them
    "artefactImportId": str(artefacts_import_id),
    "externalId": "my-byom-version-id"
}

job = model_api.import_byom(model["id"], import_request)

# wait until the job completes (if the job fails it will raise an exception)
job_api.wait(job["id"])

# now you can list the artefacts which were uploaded and linked to the model version
trained_model_id = job["metadata"]["trainedModel"]["id"]
artefacts = trained_model_artefacts_api.list_artefacts(trained_model_id)

Release Notes

6.1.0

• Cleanup: Remove all non OAuth2 (JWT) authentication methods
• Cleanup: Remove aoa configure
• Feature: Improve error messages to user on CLI
• Feature: Add aoa link for linking project to repo locally
• Bug: Don't show archived datasets
• Bug: Fix issue with aoa feature create-table

6.0.0

• Feature: Support API changes on ModelOps 06.00
• Feature: CLI DX improvements
• Feature: Add Telemetry query bands to session
• Feature: aoa feature support for managing feature metadata
• Feature: aoa add uses reference git repository for model templates
• Feature: Improve DX from Notebooks

5.0.0

• Feature: Add simpler teradataml context creation via aoa_create_context
• Feature: Add database to connections
• Feature: Support for human-readable model folder names
• Feature: Improve UX of aoa run
• Feature: Improved error messages for users related to auth and configure
• Refactor: Package refactor of aoa.sto.util to aoa.util.sto
• Bug: cli listing not filtering archived entities
• Cleanup: Remove pyspark support from CLI

4.1.12

• Bug: aoa connection add now hides password symbols
• Bug: sto.util.cleanup_cli() used hardcoded models table
• Feature: sto.util.check_sto_version() checks In-Vantage Python version compatibility
• Feature: sto.util.collect_sto_versions() fetches dict with Python and packages versions

4.1.11

• Bug: aoa run (evaluation) for R now uses the correct scoring file

4.1.10

• Bug: aoa init templates were out of date
• Bug: aoa run (score) didn't read the dataset template correctly
• Bug: aoa run (score) tried to publish to prometheus
• Bug: aoa run (score) not passing model_table kwargs

4.1.9

• Bug: Histogram buckets incorrectly offset by 1 for scoring metrics

4.1.7

• Bug: Quoted and escaped exported connection environmental variables
• Bug: aoa clone with path argument didn't create .aoa/config.yaml in correct directory
• Feature: aoa clone without path now uses repository name by default
• Feature: update BYOM import to upload artefacts before creating version

4.1.6

• Feature: Added local connections feature with Stored Password Protection
• Feature: Self creation of .aoa/config.yaml file when cloning a repo
• Bug: Fix argparse to use of common arguments
• Feature: Support dataset templates for listing datasets and selecting dataset for train/eval
• Bug: Fix aoa run for batch scoring, prompts for dataset template instead of dataset
• Bug: Fix batch scoring histograms as cumulative

4.1.5

• Bug: Fix computing stats
• Feature: Autogenerate category labels and support for overriding them
• Feature: Prompt for confirmation when retiring/archiving

4.1.4

• Feature: Retiring deployments and archiving projects support
• Feature: Added support for batch scoring monitoring

4.1.2

• Bug: Fix computing stats
• Bug: Fix local SQL model training and evaluation

4.1

• Bug: CLI shows archived entities when listing datasets, projects, etc
• Bug: Adapt histogram bins depending on range of integer types.

4.0

• Feature: Extract and record dataset statistics for Training, Evaluation

3.1.1

• Feature: aoa clone respects project branch
• Bug: support Source Model ID from the backend

3.1

• Feature: ability to separate evaluation and scoring logic into separate files for Python/R

3.0

• Feature: Add support for Batch Scoring in run command
• Feature: Added STO utilities to extract metadata for micro-models

2.7.2

• Feature: Add support for OAUTH2 token refresh flows
• Feature: Add dataset connections api support

2.7.1

• Feature: Add TrainedModelArtefactsApi
• Bug: pyspark cli only accepted old resources format
• Bug: Auth mode not picked up from environment variables

2.7.0

• Feature: Add support for dataset templates
• Feature: Add support for listing models (local and remote), datasets, projects
• Feature: Remove pykerberos dependency and update docs
• Bug: Fix tests for new dataset template api structure
• Bug: Unable to view/list more than 20 datasets / entities of any type in the cli

2.6.2

• Bug: Added list resources command.
• Bug: Remove all kerberos dependencies from standard installation, as they can be now installed as an optional feature.
• Feature: Add cli support for new artefact path formats

2.6.1

• Bug: Remove pykerberos as an installation dependency.