synapseml-mlflow 2.0.3

MLflow plugin in Microsoft Fabric platform.

Introduction

Microsoft Fabric MLflow Plugin

This is a preview version of the MLflow plugin that connects the Python MLflow library to Fabric MLflow endpoints via its entry-points registry. When the MLflow tracking URI is set to start with sds://, this plugin is activated and uses Fabric MLflow backend services as the MLflow tracking store.

Dependencies

Fabric officially supports MLflow 2.x. While support for MLflow 3 is in progress, please install MLflow or mlflow-skinny 2.x in your development or Fabric Notebook environment for now.

Setup

Fabric Notebook

%pip install synapseml-mlflow[online-notebook]

Don't forget the % prefix, which ensures the package is updated on all Spark driver and worker nodes at the same time.

In a Fabric Notebook, the MLflow tracking URI is set by default to the home Fabric workspace of your notebook. For example:

sds://api.fabric.microsoft.com/v1/workspaces/3aa8fea3-42f6-4326-924c-8ea8f9bd715f/mlflow

There is no need to set the tracking URI manually if you only want to log models to the same workspace. It is configured automatically, and private-link conditions are handled as well.

To log to a different workspace, simply change the workspace ID in the tracking URI.

import os

os.environ["MLFLOW_TRACKING_URI"] = "sds://api.fabric.microsoft.com/v1/workspaces/deeb59d0-8608-447c-98f3-09408a20cfe0/mlflow"

Other Environments

The optional online-notebook dependencies are not required:

pip install synapseml-mlflow

You must set the tracking URI first so the plugin knows your workspace ID. The URI format is the same as in a Fabric Notebook.

You must also configure credentials beforehand. Here is a complete example:

from azure.identity import ( 
    ClientSecretCredential, 
    DefaultAzureCredential, 
    DeviceCodeCredential, 
    InteractiveBrowserCredential,
) 

from fabric.analytics.environment.credentials import SetFabricAnalyticsDefaultTokenCredentialsGlobally

# Choose any credential type and register it with the Fabric Analytics SDK
SetFabricAnalyticsDefaultTokenCredentialsGlobally(InteractiveBrowserCredential())

import os

os.environ["MLFLOW_TRACKING_URI"] = "sds://api.fabric.microsoft.com/v1/workspaces/3aa8fea3-42f6-4326-924c-8ea8f9bd715f/mlflow"

import mlflow

mlflow.set_experiment("mlflow-experiment-1")

Workspace Private Link (WSPL)

If the target Fabric workspace has an inbound restriction, or your current workspace has an outbound restriction, you need to set the tracking URI host to the WSPL-specific FQDN. The format is shown below, where 3aa8fea3-42f6-4326-924c-8ea8f9bd715f is the workspace ID:

sds://3aa8fea342f64326924c8ea8f9bd715f.z3a.w.api.fabric.microsoft.com/v1/workspaces/3aa8fea3-42f6-4326-924c-8ea8f9bd715f/mlflow

Alternatively, you can let the Fabric Analytics SDK print the FQDN for you:

from fabric.analytics.environment.context import FabricContext, InternalContext

context = FabricContext(workspace_id="3aa8fea3-42f6-4326-924c-8ea8f9bd715f", internal_context=InternalContext(is_wspl_enabled=True))
print(context.pbi_shared_host)