Significance Analysis for HPO-algorithms performing on multiple benchmarks
Project description
Significance Analysis
This package is used to analyse datasets of different HPO-algorithms performing on multiple benchmarks.
Note
As indicated with the v0.x.x
version number, Significance Analysis is early stage code and APIs might change in the future.
Documentation
Please have a look at our example. The dataset should have the following format:
system_id (algorithm name) |
input_id (benchmark name) |
metric (mean/estimate) |
optional: bin_id (budget/traininground) |
---|---|---|---|
Algorithm1 | Benchmark1 | x.xxx | 1 |
Algorithm1 | Benchmark1 | x.xxx | 2 |
Algorithm1 | Benchmark2 | x.xxx | 1 |
... | ... | ... | ... |
Algorithm2 | Benchmark2 | x..xxx | 2 |
In this dataset, there are two different algorithms, trained on two benchmarks for two iterations each. The variable-names (system_id, input_id...) can be customized, but have to be consistent throughout the dataset, i.e. not "mean" for one benchmark and "estimate" for another. The conduct_analysis
function is then called with the dataset and the variable-names as parameters.
Optionally the dataset can be binned according to a fourth variable (bin_id) and the analysis is conducted on each of the bins seperately, as shown in the code example above. To do this, provide the name of the bin_id-variable, the bin intervals and the labels for thems.
Installation
Using pip
pip install significance-analysis
Using R, >=4.0.0 install packages: Matrix, emmeans, lmerTest
Usage
- Generate data from HPO-algorithms on benchmarks, saving data according to our format.
- Call function
conduct_analysis
on dataset, while specifying variable-names
In code, the usage pattern can look like this:
import pandas as pd
from signficance_analysis import conduct_analysis
# 1. Generate/import dataset
data = pd.read_csv("./significance_analysis_example/exampleDataset.csv")
# 2. Analyse dataset
conduct_analysis(data, "mean", "acquisition", "benchark")
For more details and features please have a look at our example.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for significance_analysis-0.1.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82aa90d45bd4f74b46cb26ad2c6fa69a7f34898e058df4531b082b6f9c059a49 |
|
MD5 | 47484c784eccd5253d1ac4823b3c8281 |
|
BLAKE2b-256 | a2bef246e907c1f889f2a8e98e227d9f9c49e9b5fee3b47276590b8d8ca78164 |
Hashes for significance_analysis-0.1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 764e74384ecf3f6cf4c8a7db6b7313d4c5e807365ed54338b0b8ac533e0b79a8 |
|
MD5 | 425787dcefab63e288f5c6fca35a39b7 |
|
BLAKE2b-256 | b89157d3d70d7fe34b888b68eaf1d5a25253ed2d91cf49c0e4c6c8760a0c9087 |