Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Tidy up your machine learning experiments

Project description

:dress: Outfit [WIP]

Outfit is a lightweight library to tidy up your machine learning experiments in a simple way.

The idea of Outfit is to store in your Wardrobe your parameters, output file, scores and features in order to be able to make a request and find out which are your best experimentation according to a given criterion.

How install outfit ?

PyPI:

pip install outfit

Dev version:

git clone https://github.com/AdilZouitine/outfit
cd outfit
pip install -r requirements.txt
pip install -e .

How outfit works ?

  • Tutorial 1: Build a simple model management pipeline with outfit for a CNN with Pytorch on MNIST dataset Open In Colab
import datetime

# Here import all the libraries you need for your experiment

from outfit import Wardrobe, getlog

wardrobe = Wardrobe(db_path='foo/bar/mnist.db')

exp = {
    'experiment_name': 'ResNet18',
    'comment': 'Use differential learning rate',
    'date_experiment': datetime.datetime.now()
}

wardrobe.add_experiment(**exp)

param = {
    'dropout': 0.20,
    'kernel_size': '3x3',
    'conv_block_1_lr': 0.01,
    'conv_block_2_lr': 0.001
   }

# Create the instance of your model here with your parameters
wardrobe.add_dict_parameter(param)


# Do your training phase here.
output = {'training log': '/result/training_log_resnet18.txt'}

@getlog(filepath=output['training log'])
def train_model(model, loaders, loss, lr_scheduler, n_epoch):
    ...

output.update({
    'tensorboard': '/result/event.tb',
    'model': 'diff_lr_resnet18.pth'
})

wardrobe.add_dict_output(output)

score = {
    'train acc': 0.96,
    'train loss': 0.430,
    'val acc': 0.94,
    'val loss': 0.460
     }
wardrobe.add_dict_score(score)


wardrobe.tidy() # commit your experiment in database
# If you want to get the best experiments 

for exp in wardrobe.get_best_scores(mode='max',on_score='val acc'):
    '''
    Verbose is true by default and will print on the console 
    at each iteration the parameters, output file, 
    features and scores in a table format.

    Also returns in dictionary the parameters, output file, features and scores.
    '''
    ...

Output:

════════════════════
│ TOP 1 EXPERIMENT │
════════════════════



Table : Experiment 

╒════╤═════════════════╤═══════════════════╤════════════════════════════════╤════════════════════╕
│    │   id_experiment │ experiment_name   │ comment                        │ date_experiement   │
╞════╪═════════════════╪═══════════════════╪════════════════════════════════╪════════════════════╡
│  0 │               1 │ ResNet18          │ Use differential learning rate │                    │
╘════╧═════════════════╧═══════════════════╧════════════════════════════════╧════════════════════╛


Table : Parameter 

╒════╤════════════════╤══════════════════╤═════════════╤══════════════╕
│    │   id_parameter │ parameter_name   │ parameter   │   experiment │
╞════╪════════════════╪══════════════════╪═════════════╪══════════════╡
│  0 │              1 │ dropout          │ 0.2         │            1 │
├────┼────────────────┼──────────────────┼─────────────┼──────────────┤
│  1 │              2 │ kernel_size      │ 3x3         │            1 │
├────┼────────────────┼──────────────────┼─────────────┼──────────────┤
│  2 │              3 │ conv_block_1_lr  │ 0.01        │            1 │
├────┼────────────────┼──────────────────┼─────────────┼──────────────┤
│  3 │              4 │ conv_block_2_lr  │ 0.001       │            1 │
╘════╧════════════════╧══════════════════╧═════════════╧══════════════╛


Table : Output 

╒════╤═════════════╤═══════════════╤═══════════════════════════════════╤══════════════╕
│    │   id_output │ type_output   │ path_output                       │   experiment │
╞════╪═════════════╪═══════════════╪═══════════════════════════════════╪══════════════╡
│  0 │           1 │ training log  │ /result/training_log_resnet18.txt │            1 │
├────┼─────────────┼───────────────┼───────────────────────────────────┼──────────────┤
│  1 │           2 │ tensorboard   │ /result/event.tb                  │            1 │
├────┼─────────────┼───────────────┼───────────────────────────────────┼──────────────┤
│  2 │           3 │ model         │ diff_lr_resnet18.pth              │            1 │
╘════╧═════════════╧═══════════════╧═══════════════════════════════════╧══════════════╛


Table : Score 

╒════╤════════════╤══════════════╤═════════╤══════════════╕
│    │   id_score │ type_score   │   score │   experiment │
╞════╪════════════╪══════════════╪═════════╪══════════════╡
│  0 │          1 │ train acc    │    0.96 │            1 │
├────┼────────────┼──────────────┼─────────┼──────────────┤
│  1 │          2 │ train loss   │    0.43 │            1 │
├────┼────────────┼──────────────┼─────────┼──────────────┤
│  2 │          3 │ val acc      │    0.94 │            1 │
├────┼────────────┼──────────────┼─────────┼──────────────┤
│  3 │          4 │ val loss     │    0.46 │            1 │
╘════╧════════════╧══════════════╧═════════╧══════════════╛


Table : Feature 

Other solution:

mlflow & dvc.

These solutions are great, they also offer a user interface and have many more options than my library however for a simple use where you only want to organize your experimentation and make a simple query. Both solutions seem to be overkill.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for outfit, version 0.0.2
Filename, size File type Python version Upload date Hashes
Filename, size outfit-0.0.2-py3-none-any.whl (7.4 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size outfit-0.0.2.tar.gz (7.1 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page