xanity 0.1b21

Package for managing experiments and analyses

xanity is experiment SANITY

Xanity is meant to allow easy and free creation of multiple experiments among which there is a shared codebase or system (conda) environment.

Xanity helps manage the following:

• experiment source code
• analysis source code
• experiment parameters
• experiment data
• run logs

Dependencies:

• conda (from Anaconda or my pref, Miniconda)

Usage:

xanity commands generally assume they're being called from inside a xanity project directory tree. Some commands accept additional arguments to specify the xanity proj path:

xanity init [--with-examples] [new-dir]
    < will init a new xanity project directory tree in either the $PWD or
    in the specified "new dir" with or without example files>

xanity setup [proj-dir]
    < will setup or check dependencies in the environment associated with the
    xanity project ($PWD or specified) >

xanity status [proj-dir]
    < will print the status of the current xanity project to STDOUT >

xanity run [experiment_names] [-a analyses[...]]
    < will run all (or the specified) experiments and optionally, analyses >

xanity analyse [-a analyses] [run_data_root]
    < will run all (or the specified) analyses on the most recent (or specified)
    data >

Example:

0. install xanity into a new conda env (or if you want, your system python):

   conda create -n xanity_test python=3.6
   conda activate xanity_test
   pip install xanity
   

   or for system python

   pip install --user xanity
   

1. initialize a xanity project and move inside with:

   xanity init --with-examples xanity_test_proj
   cd xanity_test_proj
   

   this will create a skeleton directory tree for your experiments and analyses which will be populated with some examples.

2. Open xanity_test_proj/experiments/experiment1.py and have a look. This is a skeletal experiment.

3. Open xanity_test_proj/analyses/analysis1.py and have a look. This is a skeletal analysis.

4. find the conda-environment.yaml file and tweak its contents to suit your requiremnts.

5. resolve these requirements (create/update a conda env) with:

   xanity setup

6. run everything with:

   xanity run -a
   

7. you will find all the experimental data organized under the data/runs directory tree. Source-code snapshots are tarred and kept with the data they produced. Logs are kept too.

8. you can run an analysis script on a completed run:

   xanity analyze experiment1

   this will look for the most recent (or specified) dir of run data, and run the analysis found at analyses/myfaveexp on that data.

9. relax. collect Nobel.

Experiment file skeleton:

(xanity-proj-root/experiments/*.py)

a. Each experiment module must have a main() function defined:

• (xanity will look for and invoke the main() function in each experiment).
• Any parameters to the experiment should be arguments to the main(). (leaving the experiment exposed in the module makes your code more portable.. I think. an alternative would be to define an experiment class inside xanity and make each experiment an instance)

b. The xanity.experiment_parameters() call registers the parameter permutations to use when running the experiment.

c. Include the xanity.metadata_hook() and xanity.run_hook() function calls. - The metadata_hook will return relevant xanity metadata (loggers, data paths, etc) - The run_hook will run the experiment if it's invoked directly as a script or as a module:

# experiments/example_experiment.py

import xanity
import numpy as np

# flag this experiment for analysis
xanity.analyze_this()

# register parameter sweeps you'd like to do
xanity.experiment_parameters({
    'n_trials': [100,150,200],
    'train_frac': [0.9, 0.5, 0.1],
    'scale': [1,2,3,]
})

# parameters the experiment will accept
def main(n_trials=200, scale =5, main_frac=0.8):

    fakevar = scale * np.random.rand(n_trials)**2
    xanity.log("here is a print from experiment 1")
    xanity.save_variable(fakevar)

if __name__=='__main__':
    xanity.run_hook()

Analysis file skeleton:

(xanity-proj-root/analyses/*.py)

a. Each analysis module must have a main() function defined:

• (xanity will look for and invoke the main() function in each analysis).
• The only parameter to the analysis is the path to the root of a run (or set of runs). (leaving the experiment exposed in the module makes your code more portable.. I think. an alternative would be to define an experiment class inside xanity and make each experiment an instance)

b. The call to xanity.analysis.link_experiments() registers the names of experiments to associate with this analysis.

c. Include the xanity.analysis.metadata_hook() and xanity.analysis.run_hook() function calls. - The metadata_hook will return relevant xanity metadata (loggers, data paths, etc) - The run_hook will run the analysis if it's invoked directly as a script or as a module

# analyses/example_analysis.py

import xanity
import matplotlib.pyplot as plt

# define which experiments to associate this analysis with
xanity.associated_experiments([
        'experiment1',
        #'experiment2',
        #'experiment3',
        ])

# the analysis takes a single argument... path of data (xanity will provide)

ef main(data_dir):

    data, paths = xanity.load_variable('fakevar')

    plt.figure()
    for d in data:
        d.sort()
        plt.plot(d)

    plt.show()

if __name__=='__main__':
    xanity.run_hook()