Machine Learning Lab

A lightweight command line interface for the management of arbitrary machine learning tasks.

Documentation is available at: https://bering-ml-lab.readthedocs.io/en/latest/

NOTE: Lab is in active development - expect a bumpy ride!

Installation

The latest stable version can be installed directly from PyPi:

pip install lab-ml

Development version can be installed from github.

git clone https://github.com/beringresearch/lab
cd lab
pip install --editable .

Concepts

Lab employs three concepts: reproducible environment, logging, and model persistence. A typical machine learning workflow can be turned into a Lab Experiment by adding a single decorator.

Creating a new Lab Project

lab init --name [NAME]

Lab will look for a requirements.txt file in the working directory to generate a portable virtual environment for ML experiments.

Setting up a Lab Experiment

Here's a simple script that trains an SVM classifier on the iris data set:

from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_score

C = 1.0
gamma = 0.7
iris = datasets.load_iris()
X = iris.data
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.24, random_state=42)

clf = svm.SVC(C, 'rbf', gamma=gamma, probability=True)
clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred, average = 'macro')

It's trivial to create a Lab Experiment using a simple decorator:

from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_score

from lab.experiment import Experiment ## New Line

e = Experiment() ## New Line

@e.start_run ## New Line
def train():
    C = 1.0
    gamma = 0.7
    iris = datasets.load_iris()
    X = iris.data
    y = iris.target

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.24, random_state=42)

    clf = svm.SVC(C, 'rbf', gamma=gamma, probability=True)
    clf.fit(X_train, y_train)

    y_pred = clf.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    precision = precision_score(y_test, y_pred, average = 'macro')

    e.log_metric('accuracy_score', accuracy) ## New Line
    e.log_metric('precision_score', precision) ## New Line

    e.log_parameter('C', C) ## New Line
    e.log_parameter('gamma', gamma) ## New Line

    e.log_model('svm', clf) ## New Line

Running an Experiment

Lab Experiments can be run as:

lab run <PATH/TO/TRAIN.py>

Comparing models

Lab assumes that all Experiments associated with a Project log consistent performance metrics. We can quickly assess performance of each experiment by running:

lab ls

Experiment    Source              Date        accuracy_score    precision_score
------------  ------------------  ----------  ----------------  -----------------
49ffb76e      train_mnist_mlp.py  2019-01-15  0.97: ██████████  0.97: ██████████
261a34e4      train_mnist_cnn.py  2019-01-15  0.98: ██████████  0.98: ██████████

Pushing models to a centralised repository

Lab experiments can be pushed to a centralised filesystem through integration with minio. Lab assumes that you have setup minio on a private cloud.

Lab can be configured once to interface with a remote minio instance:

lab config minio --tag my-minio --endpoint [URL:PORT] --accesskey [STRING] --secretkey [STRING]

To push a local lab experiment to minio:

lab push --tag my-minio --bucket [BUCKETNAME] .

Copyright 2020, Bering Limited