Skip to main content

Weights and Biases Hyperparameter Sweeps Engine.

Project description

W&B Hyperparameter Sweeps Engine

This repo contains the routines that generate hyperparameter sweep suggestions in the W&B backend and client local controller.

Issues are not enabled in this repository. Please open issues related to sweeps in the wandb client library github issues page.

Installation

To install:

pip install sweeps

Examples

Get next run in a sweep.

Requires two arguments, config, the config that defines the sweep, and runs, the other runs in the sweep

config:

{
    "metric": {"name": "loss", "goal": "minimize"},
    "method": "bayes",
    "parameters": {
        "v1": {"min": 1, "max": 10},
        "v2": {"min": 1.0, "max": 10.0},
    },
}

runs:

[
    SweepRun(
        name="b",
        state=RunState.finished,
        history=[
            {"loss": 5.0},
        ],
        config={"v1": {"value": 7}, "v2": {"value": 6}},
        summary_metrics={"zloss": 1.2},
    ),
    SweepRun(
        name="b2",
        state=RunState.finished,
        config={"v1": {"value": 1}, "v2": {"value": 8}},
        summary_metrics={"loss": 52.0},
        history=[],
    )
]

Codepath:

suggestion = next_run(config, runs)

next_run:

  • validates that sweep config conforms to the jsonschema in config/schema.json, if not, it raises a ValidationError
  • parses the config file and determines the method that it should use to find the next run (in this case bayes_search_next_run)
  • calls bayes_search_next_run(config, runs) and returns the suggested SweepRun

Return list of runs to stop in a sweep.

Requires two arguments, config, the config that defines the sweep, and runs, the other runs in the sweep

config:

{
    "method": "grid",
    "metric": {"name": "loss", "goal": "minimize"},
    "early_terminate": {
        "type": "hyperband",
        "max_iter": 5,
        "eta": 2,
        "s": 2,
    },
    "parameters": {"a": {"values": [1, 2, 3]}},
}

runs:

[
    SweepRun(
        name="a",
        state=RunState.finished,  # This is already stopped
        history=[
            {"loss": 10},
            {"loss": 9},
        ],
    ),
    SweepRun(
        name="b",
        state=RunState.running,  # This should be stopped
        history=[
            {"loss": 10},
            {"loss": 10},
        ],
    ),
    SweepRun(
        name="c",
        state=RunState.running,  # This passes band 1 but not band 2
        history=[
            {"loss": 10},
            {"loss": 8},
            {"loss": 8},
        ],
    ),
    SweepRun(
        name="d",
        state=RunState.running,
        history=[
            {"loss": 10},
            {"loss": 7},
            {"loss": 7},
        ],
    ),
    SweepRun(
        name="e",
        state=RunState.finished,
        history=[
            {"loss": 10},
            {"loss": 6},
            {"loss": 6},
        ],
    ),
]

Codepath:

to_stop = stop_runs(config, runs)

stop_runs:

  • validates that sweep config conforms to the jsonschema in config/schema.json, if not, it raises a ValidationError
  • parses the config file and determines the method that it should use to early terminate runs (in this case hyperband_stop_runs)
  • calls hyperband_stop_runs(config, runs) and returns the SweepRuns to stop

Testing

To run tests:

tox

Contributing

Install the development requirements:

pip install -r requirements.dev.txt

Install the pre-commit hooks:

pre-commit install .

PRs must:

  • Not degrade test coverage (automatically calculated via codecov)
  • Use type-hints on all public functions
  • Pass linting checks
  • Be reviewed and approved by at least 1 maintainer

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sweeps-0.1.0.tar.gz (21.9 kB view hashes)

Uploaded Source

Built Distribution

sweeps-0.1.0-py3-none-any.whl (24.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page