Weights and Biases Hyperparameter Sweeps Engine.
Project description
W&B Hyperparameter Sweeps Engine
This repo contains the routines that generate hyperparameter sweep suggestions in the W&B backend and client local controller.
Issues are not enabled in this repository. Please open issues related to sweeps in the wandb client library github issues page.
Installation
To install:
pip install sweeps
Examples
Get next run in a sweep.
Requires two arguments, config
, the config that defines the sweep, and runs
, the other runs in the sweep
config
:
{
"metric": {"name": "loss", "goal": "minimize"},
"method": "bayes",
"parameters": {
"v1": {"min": 1, "max": 10},
"v2": {"min": 1.0, "max": 10.0},
},
}
runs
:
[
SweepRun(
name="b",
state=RunState.finished,
history=[
{"loss": 5.0},
],
config={"v1": {"value": 7}, "v2": {"value": 6}},
summary_metrics={"zloss": 1.2},
),
SweepRun(
name="b2",
state=RunState.finished,
config={"v1": {"value": 1}, "v2": {"value": 8}},
summary_metrics={"loss": 52.0},
history=[],
)
]
Codepath:
suggestion = next_run(config, runs)
next_run
:
- validates that sweep config conforms to the jsonschema in
config/schema.json
, if not, it raises aValidationError
- parses the config file and determines the method that it should use to find the next run (in this case
bayes_search_next_run
) - calls
bayes_search_next_run(config, runs)
and returns the suggestedSweepRun
Return list of runs to stop in a sweep.
Requires two arguments, config
, the config that defines the sweep, and runs
, the other runs in the sweep
config
:
{
"method": "grid",
"metric": {"name": "loss", "goal": "minimize"},
"early_terminate": {
"type": "hyperband",
"max_iter": 5,
"eta": 2,
"s": 2,
},
"parameters": {"a": {"values": [1, 2, 3]}},
}
runs
:
[
SweepRun(
name="a",
state=RunState.finished, # This is already stopped
history=[
{"loss": 10},
{"loss": 9},
],
),
SweepRun(
name="b",
state=RunState.running, # This should be stopped
history=[
{"loss": 10},
{"loss": 10},
],
),
SweepRun(
name="c",
state=RunState.running, # This passes band 1 but not band 2
history=[
{"loss": 10},
{"loss": 8},
{"loss": 8},
],
),
SweepRun(
name="d",
state=RunState.running,
history=[
{"loss": 10},
{"loss": 7},
{"loss": 7},
],
),
SweepRun(
name="e",
state=RunState.finished,
history=[
{"loss": 10},
{"loss": 6},
{"loss": 6},
],
),
]
Codepath:
to_stop = stop_runs(config, runs)
stop_runs
:
- validates that sweep config conforms to the jsonschema in
config/schema.json
, if not, it raises aValidationError
- parses the config file and determines the method that it should use to early terminate runs (in this case
hyperband_stop_runs
) - calls
hyperband_stop_runs(config, runs)
and returns theSweepRun
s to stop
Testing
To run tests:
tox
Contributing
Install the development requirements:
pip install -r requirements.dev.txt
Install the pre-commit hooks:
pre-commit install .
PRs must:
- Not degrade test coverage (automatically calculated via codecov)
- Use type-hints on all public functions
- Pass linting checks
- Be reviewed and approved by at least 1 maintainer
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.