Skip to main content

Weights and Biases Hyperparameter Sweeps Engine.

Project description

Weights & Biases Weights & Biases

Sweeps: Hyperparameter Optimization Engine

This repo contains the routines that generate hyperparameter sweep suggestions in the W&B backend and client local controller.

Issues are not enabled in this repository. Please open issues related to sweeps in the wandb client library github issues page.

Installation

To install:

pip install sweeps

Examples

Get next run in a sweep.

Requires two arguments, config, the config that defines the sweep, and runs, the other runs in the sweep

config:

{
    "metric": {"name": "loss", "goal": "minimize"},
    "method": "bayes",
    "parameters": {
        "v1": {"min": 1, "max": 10},
        "v2": {"min": 1.0, "max": 10.0},
    },
}

runs:

[
    SweepRun(
        name="b",
        state=RunState.finished,
        history=[
            {"loss": 5.0},
        ],
        config={"v1": {"value": 7}, "v2": {"value": 6}},
        summary_metrics={"zloss": 1.2},
    ),
    SweepRun(
        name="b2",
        state=RunState.finished,
        config={"v1": {"value": 1}, "v2": {"value": 8}},
        summary_metrics={"loss": 52.0},
        history=[],
    )
]

Codepath:

suggestion = next_run(config, runs)

next_run:

  • validates that sweep config conforms to the jsonschema in config/schema.json, if not, it raises a ValidationError
  • parses the config file and determines the method that it should use to find the next run (in this case bayes_search_next_run)
  • calls bayes_search_next_run(config, runs) and returns the suggested SweepRun

Return list of runs to stop in a sweep.

Requires two arguments, config, the config that defines the sweep, and runs, the other runs in the sweep

config:

{
    "method": "grid",
    "metric": {"name": "loss", "goal": "minimize"},
    "early_terminate": {
        "type": "hyperband",
        "max_iter": 5,
        "eta": 2,
        "s": 2,
    },
    "parameters": {"a": {"values": [1, 2, 3]}},
}

runs:

[
    SweepRun(
        name="a",
        state=RunState.finished,  # This is already stopped
        history=[
            {"loss": 10},
            {"loss": 9},
        ],
    ),
    SweepRun(
        name="b",
        state=RunState.running,  # This should be stopped
        history=[
            {"loss": 10},
            {"loss": 10},
        ],
    ),
    SweepRun(
        name="c",
        state=RunState.running,  # This passes band 1 but not band 2
        history=[
            {"loss": 10},
            {"loss": 8},
            {"loss": 8},
        ],
    ),
    SweepRun(
        name="d",
        state=RunState.running,
        history=[
            {"loss": 10},
            {"loss": 7},
            {"loss": 7},
        ],
    ),
    SweepRun(
        name="e",
        state=RunState.finished,
        history=[
            {"loss": 10},
            {"loss": 6},
            {"loss": 6},
        ],
    ),
]

Codepath:

to_stop = stop_runs(config, runs)

stop_runs:

  • validates that sweep config conforms to the jsonschema in config/schema.json, if not, it raises a ValidationError
  • parses the config file and determines the method that it should use to early terminate runs (in this case hyperband_stop_runs)
  • calls hyperband_stop_runs(config, runs) and returns the SweepRuns to stop

Testing

Tests are run using tox, the makefile defines convenience commands for short and long tests:

make test-short
make test-full

Contributing

Review the Contributing Instructions

Install the development requirements:

pip install -r requirements.dev.txt

Install the pre-commit hooks:

pre-commit install .

Run formatting and tests:

make format
make test

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sweeps-0.2.0.tar.gz (26.0 kB view hashes)

Uploaded source

Built Distribution

sweeps-0.2.0-py3-none-any.whl (28.4 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page