glaux 0.1.3

A personal library for saving experiment results

Glaux - A personal library for saving experiments results

Introduction

Welcome to Glaux, a library for saving the results of my experiments! Glaux is a personal library designed to store the results of my experiments during my PhD. The experiments I run always follow a 3-level design:

• Level 1 - Experiment: This is the concept I want to test. It contains a set of hypotheses and a typical machine learning workflow..
• Level 2 - Settings: As in any proper experiment, I need to test different hyperparameter configurations to see how they affect the results.
• Level 3 - Trial: As in any proper experiment, I need to test different hyperparameter configurations to see how they affect the results.

Typically, for each hyperparameter configuration (settings), many trials are executed in order to compare them statistically. This is usually what an experiment consists of. Settings are typically represented as a collection of hyperparameters or metadata shared across a group of trials, while trials store the results, execution-specific metadata, and even full models if needed and if they are not too large.}

Usage

To create an experiment, the GlauxExp class is used. Settings and trials are stored as Python dictionaries. For settings, it is recommended that dictionaries contain only native Python types and NumPy arrays. Trials are more flexible and can store native types or custom class objects.

Example

Code:

# USE MODE:
import numpy as np

# Create a Glaux object
exp = GlauxExp(filename='test.pickle',description='Random number is less than a threshold')

# Create una configuration
thres = np.array([0.5, 0.1, 0.5, 0.9])

# Run the experiment
for _ in range(10):
    rnd_less = np.random.random(size=4) < thres
    exp.insert_trial( settings={'thres':thres}, 
                      trial={'result':rnd_less,'count':rnd_less.sum()} )
    
# Create a new configuration
thres = np.array([0.8, 0.8, 0.1, 0.1])

# Run the experiment again
for _ in range(10):
    rnd_less = np.random.random(size=4) < thres
    exp.insert_trial( settings={'thres':thres}, 
                      trial={'result':rnd_less} )


review_exp = GlauxExp(filename='test.pickle')
print( review_exp )

Output:

Random number is less than a threshold
id (0) | n_trials 10 | settings:{'thres': array([0.5, 0.1, 0.5, 0.9])}
id (1) | n_trials 10 | settings:{'thres': array([0.8, 0.8, 0.1, 0.1])}

Code:

# With GlauxExp().get_trial( id, keys ) you can access especific fields of the trials    
review_exp.get_trials(0, ['result','count'])

Output:

[[array([ True, False,  True,  True]), np.int64(3)],
 [array([False, False, False,  True]), np.int64(1)],
 [array([ True, False,  True,  True]), np.int64(3)],
 [array([ True, False, False,  True]), np.int64(2)],
 [array([False, False,  True,  True]), np.int64(2)],
 [array([False, False,  True,  True]), np.int64(2)],
 [array([ True, False, False,  True]), np.int64(2)],
 [array([False, False,  True,  True]), np.int64(2)],
 [array([ True, False, False, False]), np.int64(1)],
 [array([False, False,  True,  True]), np.int64(2)]]

Code:

# If keys are not especified, you access all the trials for that setting id
review_exp.get_trials(0)