hydra-orion-sweeper 1.5.0

Hydra Orion Sweeper plugin

Provides a mechanism for Hydra applications to use Orion algorithms for the optimization of the parameters of any experiment.

See website for more information

Install

pip install hydra-orion-sweeper

Search Space

Orion defines 5 different dimensions that can be used to define your search space.

• uniform(low, high, [discrete=False, precision=4, shape=None, default_value=None])

• loguniform(low, high, [discrete=False, precision=4, shape=None, default_value=None])

• normal(loc, scale, [discrete=False, precision=4, shape=None, default_value=None])

• choices(*options)

• fidelity(low, high, base=2)

Fidelity is a special dimension that is used to represent the training time, you can think of it as the epoch dimension.

Documentation

For in-depth documentation about the plugin and its configuration options you should refer to Orion as the plugin configurations are simply passed through.

• algorithm

• worker

• storage

• parametrization

Example

Configuration

defaults:
- override hydra/sweeper: orion

hydra:
    sweeper:
       params:
          a: "uniform(0, 1)"
          b: "uniform(0, 1)"

       orion:
          name: 'experiment'
          version: '1'

       algorithm:
          type: random
          config:
             seed: 1

       worker:
          n_workers: -1
          max_broken: 3
          max_trials: 100

       storage:
          type: legacy
          database:
             type: pickleddb
             host: 'database.pkl'

# Default values
a: 0
b: 0

Note

If the orion database path is relative; orion will create one database per multirun, this is because hydra is in charge of the working directory, and hydra creates a new directory per run.

To share a database between multiruns you can use an absolute path.

Code

import hydra
from omegaconf import DictConfig

@hydra.main(config_path=".", config_name="config")
def main(cfg: DictConfig) -> float:
   """Simple main function"""
   a = cfg.a
   b = cfg.b

   return float(a + b)

if __name__ == "__main__":
   main()

Running

To run the hyper parameter optimization process you need to specify the --multirun argument.

python my_app.py --multirun

The search space can also be tweaked from the command line

python my_app.py --multirun batch_size=4,8,12,16 optimizer.name=Adam,SGD 'optimizer.lr="loguniform(0.001, 1.0)"'

Note

When specifying overrides you need to be careful with your bash/zsh/fish environment and escape the arguments correctly.