pai-mlflow-test 0.0.1

Alibaba Cloud PAI MLflow Plugin

AlibabaCloud PAI MLflow Plugin

English | 中文

What is MLflow

Mlflow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry.

AlibabaCloud PAI MLflow Plugin integrated MLflow capacities with AlibabaCloud Machine Learning Platform for AI. You can manage your machine learning task by using MLflow SDK.

Installing

pip install pai-mlflow

Usage

AlibabaCloud PAI MLflow Plugin including 3 main components：PaiTrackingStore、PaiModelRegistryStore and OssArtifactRepository.

PaiTrackingStore tracking and recording machine learning tasks，logging parameters, metrics and artifacts。

PaiModelRegistryStore registering generated model, and offering model version management。

OssArtifactRepository storing artifacts via AlibabaCloud Object Storage Service

PaiTrackingStore & PaiModelRegistryStore

Set AccessKey via environment

export PAI_ACCESS_KEY_ID=LTAIxxxxxxxxxxxxxx;
export PAI_ACCESS_KEY_SECRET=asdfxxxxxxxxxxxxxxxxxxxxxxxxxxxx;

More about AccessKey, please reference create an AccessKey pair

Set MLflow tracking uri to PAI tracking uri.

import mlflow

mlflow.set_tracking_uri(
    "pai://mlflow.{regionId}.aliyuncs.com")  # replace {regionId} with the real one, likes 'cn-hangzhou'

Start training

import mlflow

"""
some pre-conditions code...
"""
with mlflow.start_run():
    mlflow.log_param("param1", randint(0, 100))
    """
    some training code...
    """

    mlflow.log_metric("foo", random())
    mlflow.log_metric("foo", random() + 1)
    mlflow.log_metric("foo", random() + 2)

More about MLflow API，please reference MLflow Docs

View Results

TODO

OssArtifactRepository

Set AccessKey via environment

You can skip this step if you already set AccessKey before using PaiTrackingStore. Try code below if you want to specify an AccessKey differ from PaiTrackingStore。

# AccessKey used by OssArtifactRepository
export MLFLOW_OSS_KEY_ID=LTAIxxxxxxxxxxxxxx;
export MLFLOW_OSS_KEY_SECRET=asdfxxxxxxxxxxxxxxxxxxxxxxxxxxxx;

# AccessKey used by PaiTrackingStore and PaiModelRegistryStore
#export PAI_ACCESS_KEY_ID=LTAIxxxxxxxxxxxxxx;
#export PAI_ACCESS_KEY_SECRET=asdfxxxxxxxxxxxxxxxxxxxxxxxxxxxx;

Save Artifacts

import mlflow

uri = "oss://your-bucket.oss-cn-hangzhou.aliyuncs.com/path/to/artifact"

exp_id = mlflow.create_experiment("exp_name", artifact_location=uri)
"""
some pre-conditions code...
"""
with mlflow.start_run(experiment_id=exp_id):
    """
    some training code...
    """
    mlflow.log_artifact("/local_path/to/artifact")

Check the artifact from OSS console