Skip to main content

Lab: a command line interface for the management of arbitrary machine learning tasks.

Project description

Documentation Status

Machine Learning Lab

A lightweight command line interface for the management of arbitrary machine learning tasks.

Documentation is available at: https://bering-ml-lab.readthedocs.io/en/latest/

NOTE: Lab is in active development - expect a bumpy ride!

alt text

Installation

The latest stable version can be installed directly from PyPi:

pip install lab-ml

Development version can be installed from github.

git clone https://github.com/beringresearch/lab
cd lab
pip install --editable .

Concepts

Lab employs three concepts: reproducible environment, logging, and model persistence. A typical machine learning workflow can be turned into a Lab Experiment by adding a single decorator.

Creating a new Lab Project

lab init --name [NAME]

Lab will look for a requirements.txt file in the working directory to generate a portable virtual environment for ML experiments.

Setting up a Lab Experiment

Here's a simple script that trains an SVM classifier on the iris data set:

from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_score

C = 1.0
gamma = 0.7
iris = datasets.load_iris()
X = iris.data
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.24, random_state=42)

clf = svm.SVC(C, 'rbf', gamma=gamma, probability=True)
clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred, average = 'macro')

It's trivial to create a Lab Experiment using a simple decorator:

from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_score

from lab.experiment import Experiment ## New Line

e = Experiment() ## New Line

@e.start_run ## New Line
def train():
    C = 1.0
    gamma = 0.7
    iris = datasets.load_iris()
    X = iris.data
    y = iris.target

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.24, random_state=42)

    clf = svm.SVC(C, 'rbf', gamma=gamma, probability=True)
    clf.fit(X_train, y_train)

    y_pred = clf.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    precision = precision_score(y_test, y_pred, average = 'macro')

    e.log_metric('accuracy_score', accuracy) ## New Line
    e.log_metric('precision_score', precision) ## New Line

    e.log_parameter('C', C) ## New Line
    e.log_parameter('gamma', gamma) ## New Line

    e.log_model('svm', clf) ## New Line

Running an Experiment

Lab Experiments can be run as:

lab run <PATH/TO/TRAIN.py>

Comparing models

Lab assumes that all Experiments associated with a Project log consistent performance metrics. We can quickly assess performance of each experiment by running:

lab ls

Experiment    Source              Date        accuracy_score    precision_score
------------  ------------------  ----------  ----------------  -----------------
49ffb76e      train_mnist_mlp.py  2019-01-15  0.97: ██████████  0.97: ██████████
261a34e4      train_mnist_cnn.py  2019-01-15  0.98: ██████████  0.98: ██████████

Pushing models to a centralised repository

Lab experiments can be pushed to a centralised filesystem through integration with minio. Lab assumes that you have setup minio on a private cloud.

Lab can be configured once to interface with a remote minio instance:

lab config minio --tag my-minio --endpoint [URL:PORT] --accesskey [STRING] --secretkey [STRING]

To push a local lab experiment to minio:

lab push --tag my-minio --bucket [BUCKETNAME] .

Copyright 2020, Bering Limited

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lab-ml-0.83.tar.gz (14.2 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page