Wandb Carbs

Wandb Carbs is a Python package that integrates the CARBS (Cost-Aware Bayesian Search) hyperparameter optimization library with Weights & Biases (W&B) sweeps. It enables cost-aware Bayesian parameter search using W&B's sweep functionality, allowing for efficient hyperparameter optimization across multiple parallel agents while storing all state within W&B.

Table of Contents

• Features
• Installation
• Usage
  • How It Works
  • Creating a Sweep
  • Running an Experiment
• Examples
• Notes
• References
• License

Features

• Cost-Aware Bayesian Optimization: Optimize hyperparameters considering both performance and computational cost.
• Seamless W&B Integration: Leverage W&B sweeps for distributed hyperparameter optimization with state stored in W&B.
• Parallel Execution: Support multiple parallel agents running as part of the sweep.
• Automatic State Management: Each agent initializes with observations from completed runs and generates suggestions for incomplete runs.
• Customizable Parameter Spaces: Support for LogSpace, LogitSpace, and LinearSpace parameter spaces.

Installation

You can install Wandb Carbs via pip:

pip install wandb_carbs

Usage

Prerequisites

• An account on Weights & Biases.
• Basic understanding of CARBS for Bayesian optimization.
• Familiarity with W&B sweeps.

How It Works

CARBS performs cost-aware Bayesian parameter search but doesn't natively integrate with W&B sweeps. Wandb Carbs bridges this gap by:

• Creating a W&B Sweep: Converts CARBS parameter definitions into a W&B sweep configuration.
• Initializing Agents: Each sweep agent run creates a new CARBS instance.
• State Initialization: Agents initialize with observations from completed runs in the sweep and generate suggestions for incomplete runs.
• Parameter Suggestion: Generates a suggestion for the current run and updates the run's configuration with the suggested parameters.
• Recording Results: After the run completes, it stores the CARBS objective and cost in the run's metadata.

This setup allows multiple parallel agents to run as part of the sweep while storing all the state in W&B.

Creating a Sweep

First, define your parameter spaces using CARBS:

from carbs import Param, LogSpace, LinearSpace

# Define parameter spaces
params = [
    Param(name='learning_rate', space=LogSpace(min=1e-5, max=1e-1)),
    Param(name='batch_size', space=LinearSpace(min=16, max=128, is_integer=True)),
]

Use the create_sweep function from Wandb Carbs to create a W&B sweep:

from wandb_carbs import create_sweep

sweep_id = create_sweep(
    sweep_name='My CARBS Sweep',
    wandb_entity='your_wandb_entity',
    wandb_project='your_wandb_project',
    carbs_spaces=params
)

This function converts the CARBS parameters into a W&B sweep configuration and returns a sweep_id that you can use to manage the sweep.

Running an Experiment

In your training script, integrate Wandb Carbs:

import wandb
from wandb_carbs import WandbCarbs
from carbs import CARBS

def train():
    # Initialize W&B run
    wandb.init()

    # Initialize CARBS
    carbs = CARBS(params=params)

    # Initialize Wandb Carbs
    wandb_carbs = WandbCarbs(carbs=carbs)

    # Get hyperparameters suggestion
    suggestion = wandb_carbs.suggest()

    # Your training code here
    # The suggested parameters are in wandb.config
    model = build_model(wandb.config)
    performance = evaluate_model(model)

    # Record observation
    objective = performance['accuracy']  # The metric you aim to optimize
    cost = performance['training_time']  # Computational cost measure
    wandb_carbs.record_observation(objective=objective, cost=cost)

    # Finish the run
    wandb.finish()

if __name__ == '__main__':
    train()

Note: The suggested hyperparameters are automatically stored in wandb.config by the W&B agent; therefore, you don't need to manually update wandb.config with the suggestion.

Examples

Full Example

import wandb
from wandb_carbs import WandbCarbs, create_sweep
from carbs import CARBS, Param, LogSpace, LinearSpace

# Define parameter spaces
params = [
    Param(name='learning_rate', space=LogSpace(min=1e-5, max=1e-1)),
    Param(name='batch_size', space=LinearSpace(min=16, max=128, is_integer=True)),
]

# Create a sweep
sweep_id = create_sweep(
    sweep_name='My CARBS Sweep',
    wandb_entity='your_wandb_entity',
    wandb_project='your_wandb_project',
    carbs_spaces=params
)

def train():
    # Initialize W&B run
    wandb.init()

    # Initialize CARBS
    carbs = CARBS(params=params)

    # Initialize Wandb Carbs
    wandb_carbs = WandbCarbs(carbs=carbs)

    # Get hyperparameters suggestion
    suggestion = wandb_carbs.suggest()

    # Your training code here
    # The suggested parameters are in wandb.config
    model = build_model(wandb.config)
    performance = evaluate_model(model)

    # Record observation
    objective = performance['accuracy']
    cost = performance['training_time']
    wandb_carbs.record_observation(objective=objective, cost=cost)

    # Finish the run
    wandb.finish()

if __name__ == '__main__':
    train()

Managing the Sweep

• Use the sweep_id returned by create_sweep to monitor and manage your sweep in the W&B dashboard.
• Ensure that all agents (parallel runs) are correctly configured to run the train function.

Notes

• Replace your_wandb_entity and your_wandb_project with your actual W&B entity and project names.
• The objective should be the metric you aim to optimize (e.g., accuracy, F1 score).
• The cost can be any measure of computational cost, like training time, memory usage, or FLOPs.
• Ensure that your build_model and evaluate_model functions use the parameters from wandb.config.

References

• CARBS: For more detailed instructions on CARBS, visit the CARBS GitHub repository.
• Weights & Biases: Learn more about W&B sweeps here.

License

This project is licensed under the MIT License - see the LICENSE file for details.