Package for managing experiments and analyses
Project description
xanity is a tool for experimentation
Xanity is meant to allow easy and free creation of multiple experiments which share some codebase similarity.
Xanity helps manage the following:
- experiment source code
- analysis source code
- experiment parameters
- experiment data
- run logs
Usage:
-
initialize $PWD (or create a new xanity directory) with:
xanity init [new_directory]
this will create a skeleton directory tree for your experiments and analyses.
-
find the conda-environment.yaml file and tweak its contents to suit your requiremnts.
-
resolve these requirements (create/update a conda env) with:
xanity setup
-
create/edit experiments inside xanity-project-root/experiments (see below for experiment file skeleton):
nano experiments/myfaveexp.py
-
run experiments (individually or collectively):
xanity run
this will run everythingxanity run myfaveexp
this will run the exp found inside experiments/myfaveexp.pyxanity run myfaveexp experiment1
this will run the two experiments givenpython -m experiment/myfaveexp
this is an example of running an experiment directlypython experiment/myfaveexp
this is another example of running an experiment directly
-
you will find all the experimental data organized under the data/runs directory tree. Source-code snapshots are tarred and kept with the data they produced. Logs are kept too.
-
you can run an analysis script on a completed run:
xanity [analyze|anal] myfaveexp [-d run_data_root_dir]
this will look for the most recent (or specified) dir of run data, and run the analysis found at analyses/myfaveexp on that data.
-
relax. collect Nobel.
Experiment file skeleton:
(xanity-proj-root/experiments/*.py)
a. Each experiment module must have a main() function defined:
- (xanity will look for and invoke the main() function in each experiment).
- Any parameters to the experiment should be arguments to the main(). (leaving the experiment exposed in the module makes your code more portable.. I think. an alternative would be to define an experiment class inside xanity and make each experiment an instance)
b. The xanity.experiment.parameters()
call registers the parameter permutations to use when running the experiment.
c. Include the xanity.metadata_hook()
and xanity.run_hook()
function calls.
- The metadata_hook will return relevant xanity metadata (loggers, data paths, etc)
- The run_hook will run the experiment if it's invoked directly as a script or as a module:
# experiments/example_experiment.py
import xanity
import numpy as np
# flag this experiment for analysis
xanity.analyze_this()
# register parameter sweeps you'd like to do
xanity.experiment_parameters({
'n_trials': [100,150,200],
'train_frac': [0.9, 0.5, 0.1],
'scale': [1,2,3,]
})
# parameters the experiment will accept
def main(n_trials=200, scale =5, main_frac=0.8):
fakevar = scale * np.random.rand(n_trials)**2
xanity.log("here is a print from experiment 1")
xanity.save_variable(fakevar)
if __name__=='__main__':
xanity.run_hook()
Analysis file skeleton:
(xanity-proj-root/analyses/*.py)
a. Each analysis module must have a main() function defined:
- (xanity will look for and invoke the main() function in each analysis).
- The only parameter to the analysis is the path to the root of a run (or set of runs). (leaving the experiment exposed in the module makes your code more portable.. I think. an alternative would be to define an experiment class inside xanity and make each experiment an instance)
b. The call to xanity.analysis.link_experiments() registers the names of experiments to associate with this analysis.
c. Include the xanity.analysis.metadata_hook()
and xanity.analysis.run_hook()
function calls.
- The metadata_hook will return relevant xanity metadata (loggers, data paths, etc)
- The run_hook will run the analysis if it's invoked directly as a script or as a module
# analyses/example_analysis.py
import xanity
import matplotlib.pyplot as plt
# define which experiments to associate this analysis with
xanity.associated_experiments([
'experiment1',
#'experiment2',
#'experiment3',
])
# the analysis takes a single argument... path of data (xanity will provide)
ef main(data_dir):
data, paths = xanity.load_variable('fakevar')
plt.figure()
for d in data:
d.sort()
plt.plot(d)
plt.show()
if __name__=='__main__':
xanity.run_hook()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.