auger.ai 0.2.5

Auger python and command line interface package

Install

pip install auger.ai

Auger.ai

Auger Cloud python and command line interface

CLI commands

• auth - allows to login into Auger Cloud

  • login
  • logout
  • whoami

• new - creates local folder for your Project and puts there auger.yaml; auger.yaml provides local context for the Project and keeps settings for Experiment(s)

• project

  • list - list all Projects for your Organization.
  • select - selects existing Project and stores it's name in auger.yaml; all further operations with DataSet(s), Experiment(s), and Model(s) will be performed in context of this Project.
  • create - creates Project on Auger Cloud; Project name will be stored in auger.yaml; all further operations with DataSet(s), Experiment(s), and Model(s) will be performed in context of this Project.
  • delete - deletes Project on Auger Cloud and clears Project name from auger.yaml
  • start - starts Project cluster.
  • stop - stops Project cluster.

• dataset

  • list - list all DataSets(s) for the Project.
  • select - selects existing DataSet and stores it's name in auger.yaml; all further operations with Experiments and Models will be performed using this DataSet.
  • create - creates new DataSet on Auger Cloud from the local or remote data file; name of the DataSet will be stored in auger.yaml; all further operations with Experiments and Models will be performed using this DataSet.
  • download - Downloads source data form Data Set on the Auger Cloud. If Data Set name is not specified on command line, auger.yaml/dataset will be used instead.
  • delete - deletes DataSet on Auger Cloud and clears DataSet name from auger.yaml

• experiment

  • list - list all Experiment(s) for the DataSet
  • start - starts Experiment with selected DataSet; Experiment settings configured in auger.yaml
  • stop - stops running experiment.
  • leaderboard - shows leaderboard of the currently running or the last completed experiment.
  • history - shows history (leaderboards and settings) of the previous experiment runs.

• model

  • list - lists all deployed models on Auger Cloud; auger.ai don't keep track of locally deployed models.
  • deploy - deploys selected model locally or on Auger Cloud.
  • predict - predicts using deployed model.

Auger.ai API

Base Classes

auger.api.Context

Context provides environment to run Auger Experiments and Models:

• loads Auger Credentials and initializes Auger REST API to communicate with remote Auger Cloud;
• loads Auger settings from auger.yaml and provides access to these settings to Auger classes and business objects;
• provides logging interface to all Auger classes and business objects.

Credentials could be acquired using Auger CLI auth command or loaded from Auger website. Credentials lookup and loading order:

• form environment variable AUGER_CREDENTIALS set with content of the credentials json;
• from auger.json file, path to folder with credentials set with environment variable AUGER_CREDENTIALS_PATH;
• from auger.json file, path to folder with credentials set with path_to_credentials key in auger.yaml
• if none above, form $HOME/.augerai/auger.json

auger.api.Project

Project provides interface to Auger Project.

• Project(context, project_name) - constructs Project instance.

  • context - instance of auger.api.Context.
  • project_name - name of the existing or new Project, optional.

• list() - lists all Projects in your Organization. Returns iterator where each item is dictionary with Project properties. Throws exception if can't validate credentials or network connection error.

  Example:

  ctx = Context()
  for project in iter(Project(ctx).list()):
    ctx.log(project.get('name'))
  

• create() - creates Project on Auger Cloud. Throws exception if can't validate credentials, Project with such name already exists, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, new_project_name).create()
  

• delete() - deletes Project on Auger Cloud. Throws exception if can't validate credentials, Project with such name doesn't exist, or network connection error.

  Example:

  ctx = Context()
  Project(ctx, existing_project_name).delete()
  

• start() - starts Project cluster. DataSet processing, Experiment runs and Model deploy and predict need cluster to perform operations and will start cluster automatically. It is possible, but not necessary, to start cluster beforehand. Throws exception if can't validate credentials or network connection error.

  Project cluster configuration defined in auger.yaml:

  cluster:
    # Cluster node type: standard|high_memory
    type: high_memory
    # Minimal number of cluster nodes
    min_nodes: 2
    # Maximum number of cluster nodes
    max_nodes: 4
    # Cluster software stack version - optional
    stack_version: experimental
  

  Example:

  ctx = Context()
  Project(ctx, project_name).start()
  

• stop() - stops Project cluster. DataSet processing, Experiment runs and Model deploy and predict need cluster to perform operations and will start cluster automatically. Cluster will stop automatically after some inactivity period. To stop it explicitly, use Project stop() method. Throws exception if can't validate credentials, such project doesn't exist, or network connection error.

  Example:

  ctx = Context()
  Project(ctx, project_name).stop()
  

• properties() - returns dictionary with Project properties. Throws exception if can't validate credentials, such Project doesn't exist, or network connection error.

  Example:

  ctx = Context()
  properties = Project(ctx, project_name).properties()
  

auger.api.DataSet

DataSet for training on Auger Cloud.

• DataSet(context, project, dataset_name) - constructs DataSet instance.

  • context - instance of auger.api.Context.
  • project - instance of auger.api.Project pointing to existing remote project.
  • dataset_name - name of the existing or new DataSet, optional.

• list() - lists all DataSets(s) for the Project. Returns iterator where each item is dictionary with DataSet properties. Throws exception if can't validate credentials, parent project doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  for dataset in iter(DataSet(ctx, project).list()):
    ctx.log(dataset.get('name'))
  

• create(source) - creates new DataSet on Auger Cloud from the local or remote data file. If dataset_name is not set, name will be selected automatically. Throws exception if can't validate credentials, parent project doesn't exist, DataSet with specified name already exists, or network connection error.

  • source - path to local or link to remote .csv or .arff file

  If Project cluster is not running, it will be started automatically to parse and preprocess DataSet.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  dataset = DataSet(ctx, project).create('../iris.csv')
  ctx.log('Created dataset %s' % dataset.name)
  

• delete() - deletes DataSet on Auger Cloud. Throws exception if can't validate credentials, parent project doesn't exist, DataSet with specified name doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  DataSet(ctx, project, dataset_name).delete()
  ctx.log('Deleted dataset %s' % dataset_name)
  

• properties() - returns dictionary with DataSet properties. Throws exception if can't validate credentials, such DataSet doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  properties = DataSet(ctx, project, dataset_name).properties()
  

auger.api.Experiment

Experiment searches for the best Model(s) for a given DataSet.

• Experiment(context, dataset, experiment_name) - constructs Experiment instance.

  • context - instance of auger.api.Context.
  • dataset - instance of auger.api.DataSet pointing to existing remote DataSet which will be used to search for the best Model.
  • experiment_name - name of the existing or new Experiment, optional.

• list() - list all Experiment(s) for the DataSet. Returns iterator where each item is dictionary with Experiment properties. Throws exception if can't validate credentials, parent DataSet doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  dataset = DataSet(ctx, project, dataset_name)
  for exp in iter(Experiment(ctx, dataset).list()):
    ctx.log(exp.get('name'))
  

• start() - starts Experiment with selected DataSet; Experiment settings configured in auger.yaml. If experiment_name is not set in constructor, unique name for the Experiment will be created automatically. Throws exception if can't validate credentials, parent DataSet doesn't exist, experiment with such name already exists, or network connection error.

  If Project cluster is not running, it will be started automatically to process search for the best Model.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  dataset = DataSet(ctx, project, dataset_name)
  experiment_name, session_id = Experiment(ctx, dataset).start()
  

  Example of the Experiment settings in auget.yaml:

  # List of columns to be excluded from the training data
  exclude:
  
  experiment:
    # Time series feature. If Data Source contains more then one DATETIME feature
    # you will have to explicitly specify feature to use as time series
    time_series:
    # List of columns which should be used as label encoded features
    label_encoded: []
    # Number of folds used for k-folds validation of individual trial
    cross_validation_folds: 5
    # Maximum time to run experiment in minutes
    max_total_time: 60
    # Maximum time to run individual trial in minutes
    max_eval_time: 1
    # Maximum trials to run to complete experiment
    max_n_trials: 10
    # Try to improve model performance by creating ensembles from the trial models
    use_ensemble: true
    ### Metric used to build Model
    # Score used to optimize ML model.
    # Supported scores for classification: accuracy, f1_macro, f1_micro, f1_weighted, neg_log_loss, precision_macro, precision_micro, precision_weighted, recall_macro, recall_micro, recall_weighted
    # Supported scores for binary classification: accuracy, average_precision, f1, f1_macro, f1_micro, f1_weighted, neg_log_loss, precision, precision_macro, precision_micro, precision_weighted, recall, recall_macro, recall_micro, recall_weighted, roc_auc, cohen_kappa_score, matthews_corrcoef
    # Supported scores for regression and time series: explained_variance, neg_median_absolute_error, neg_mean_absolute_error, neg_mean_squared_error, neg_mean_squared_log_error, r2, neg_rmsle, neg_mase, mda, neg_rmse
    metric: f1_macro
  

• stop() - stops running Experiment. Returns True is Experiment was running and stopped now, False is Experiment wasn't running and stop command was ignored. Throws exception if can't validate credentials, parent DataSet doesn't exist, Experiment with such name doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  dataset = DataSet(ctx, project, dataset_name)
  if Experiment(self.ctx, dataset, experiment_name).stop():
      ctx.log('Search is stopped...')
  else:
      ctx.log('Search is not running. Stop is ignored.')
  

• leaderboard(run_id) - leaderboard of the currently running or previously completed experiment(s). If run_id is not specified, method returns currently running or last completed experiment leaderboard; otherwise returns leaderboard for the run with specified id. Returns None if leaderboard wasn't found.

  In addition, returns status of the Experiment run:

  • preprocess - Search is preprocessing data for traing;
  • started - Search is in progress;
  • completed - Search is completed;
  • interrupted - Search was interrupted.

  Throws exception if can't validate credentials, parent DataSet doesn't exist, Experiment with such name doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  dataset = DataSet(ctx, project, dataset_name)
  # latest experiment leaderboard and latest experiment status
  leaderboard, status = Experiment(ctx, dataset, experiment_name).leaderboard()
  

• history() - history (leaderboards and settings) of the previous experiment runs. Returns iterator where each item is dictionary with properties of the previous Experiment runs. Throws exception if can't validate credentials, parent DataSet doesn't exist, Experiment with such name doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  dataset = DataSet(ctx, project, dataset_name)
  for run in iter(Experiment(self.ctx, dataset, experiment_name).history()):
      ctx.log("run id: {}, start tiem: {}, status: {}".format(
        run.get('id'),
        run.get('model_settings').get('start_time'),
        run.get('status')))
  

• properties() - returns dictionary with Experiment properties. Throws exception if can't validate credentials, such Experiment doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  dataset = DataSet(ctx, project, dataset_name)
  properties = Experiment(self.ctx, dataset, experiment_name).properties()
  

• delete() - deletes Experiment. Throws exception if can't validate credentials, such Experiment doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  dataset = DataSet(ctx, project, dataset_name)
  Experiment(self.ctx, dataset, experiment_name).delete()
  

auger.api.Model

Deploys or predicts using one of the Models from the Project Experiment(s) leaderboards.

• Model(context, project) - constructs Model instance.

  • context - instance of auger.api.Context.
  • project - instance of auger.api.Project pointing to existing remote Project.

• list() - lists all deployed models for a Project; auger.ai don't keep track of locally deployed models. Returns iterator where each item is dictionary with deployed Model properties. Throws exception if can't validate credentials or network connection error.

• deploy(model_id, locally) - deploys selected model locally or on Auger Cloud. Returns deployed model id.

  • model_id - id of the model from the any Experiment leaderboard
  • locally - deploys model locally if True, on Auger Cloud if False; optional, default is False.

  Throws exception if can't validate credentials, project of model doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  # deploys model locally
  Model(ctx, project).deploy(model_id, True)
  

• predict(filename, model_id, threshold, locally) - predicts using deployed model. Predictions stored next to the file with data to be predicted on; file name will be appended with suffix _predicted.

  • filename - file with data to be predicted
  • model_id - id of the deployed model
  • threshold - prediction threshold
  • locally - if True predict using locally deployed model, predict using model deployed on Auger Cloud

  Throws exception if can't validate credentials, project of model doesn't exist, or network connection error.

  Example:

  ctx = Context()
  project = Project(ctx, project_name)
  # predict on the locally deployed model
  Model(ctx, project).predict('../irises.csv', model_id, None, True)
  # result will be stored in ../irises_predicted.csv
  

Development Setup

We strongly recommend to install Python virtual environment:

$ pip install virtualenv virtualenvwrapper

Clone Auger Cloud repo:

$ git clone https://github.com/deeplearninc/auger-ai

Setup dependencies and Auger command line:

$ pip install -e .[all]

Running tests and getting test coverage:

$ pytest --cov='auger' --cov-report html tests/