PhaseSweep

PhaseSweep runs phase-chained hyperparameter sweeps from one ordinary YAML file. That file contains your trainer configuration, the metric, and the search plan. PhaseSweep materializes the configured trainer input for every trial, decides what to try next, persists each phase winner, and carries selected values forward as fixed inputs to later phases.

This is useful when a full joint sweep is too expensive or hard to interpret. For example, choose architecture depth, then tune learning rate, then regularization. The configuration guide explains the inheritance model and its tradeoffs.

How it works

The starter generated by phasesweep init searches depth before learning rate (abridged; the full file also sets storage, the working directory, and the metric extractor):

experiment: phasesweep_starter

# The trainer receives one complete generated YAML per trial.
trial_command: "python -m phasesweep.examples.fake_train --config {config_path}"

trainer_config:
  model:
    n_layers: 8
    dropout: 0.1
  optimizer:
    lr: 0.0003
    weight_decay: 0.05

metric:
  name: eval_loss
  goal: minimize

phases:
  - name: depth
    n_trials: 2
    sampler: { type: grid }
    search_space:
      model.n_layers: { type: categorical, choices: [6, 8] }

  - name: learning_rate
    inherits: [depth] # the winning model.n_layers is fixed here
    n_trials: 2
    sampler: { type: grid }
    search_space:
      optimizer.lr: { type: categorical, choices: [0.0001, 0.0003] }

The packaged fake trainer uses the optional report_objective(...) envelope helper. Your own trainer can report through W&B, ordinary JSON, or logs without importing PhaseSweep.

Install and try it

Requirements: Python 3.11+ and a POSIX host for real runs. GPUs are optional.

python -m pip install phasesweep
phasesweep --version

mkdir phasesweep-demo && cd phasesweep-demo
phasesweep init                       # writes the starter experiment.yaml
phasesweep validate experiment.yaml   # checks it without launching anything
phasesweep run experiment.yaml        # four tiny trials, a few seconds

phasesweep init never overwrites an existing file; use -o PATH for another destination.

The run log shows the phase chaining directly (abridged):

[depth/trial_0] python -m phasesweep.examples.fake_train --config .../trainer_config.yaml
[depth/trial_1] python -m phasesweep.examples.fake_train --config .../trainer_config.yaml
phase=depth WINNER trial=0 metric=0.3 params={'model.n_layers': 8}
[learning_rate/trial_0] python -m phasesweep.examples.fake_train --config .../trainer_config.yaml
[learning_rate/trial_1] python -m phasesweep.examples.fake_train --config .../trainer_config.yaml
phase=learning_rate WINNER trial=0 metric=0.3 params={'optimizer.lr': 0.0003}

The depth winner's model.n_layers: 8 is injected into every learning_rate trial's complete YAML. Winners persist in the working directory, so you can inspect them any time (abridged):

phasesweep show-winners experiment.yaml

phase: learning_rate
metric:
  eval_loss: 0.3
  goal: minimize
trial_number: 0
params:
  optimizer.lr: 0.0003
effective_overrides:
  model.n_layers: 8
  optimizer.lr: 0.0003
# ... completion state, fingerprints, objective provenance, and confound
# verdicts on the ranked population follow

The starter uses storage: auto, so its ledger and outputs live together under <workdir>/<experiment>/. The ledger is an Optuna journal: phasesweep status prints its path, and Optuna's own CLI and dashboard can read it. See runtime behavior for state layout, current-format requirements, locks, cleanup, GPU isolation, and resume.

Use your own trainer

Choose the input interface your trainer already accepts:

For ordinary JSON output, pass an output path under {trial_dir} to your trainer and select it with extractor: {type: json, path: result.json, key: eval.loss}. For W&B, use extractor: {type: wandb, entity: YOUR_ENTITY, project: YOUR_PROJECT, metric_key: eval/loss}, have the trainer honor the supplied W&B identity, and install the extra:

python -m pip install "phasesweep[wandb]"

metric.goal ranks each trial's selected scalar. Log matching selects the last observation by default; W&B selects an exact finished-summary key. The trainer owns evaluation, early stopping, checkpoint selection, and aggregation. Inheritance transfers parameters, not model weights.

See the configuration guide for input types, reporting, credentials, and selection details. The Tiny Decoder Enwik8 example is a complete real-trainer integration.

Connect an agent

The optional MCP server connects an AI agent to experiments you have approved without exposing config or storage paths, trainer commands, environment values, or raw logs. Sampled winner values follow the catalog's visible_params policy. Follow the MCP setup to install the extra, review the catalog authority boundary, configure a client-owned connection, and verify the result.

Reference

• Configuration guide: Experiment YAML, search spaces, inheritance, gates, and objective extractors.
• Configuration reference: per-key types, defaults, valid values, interactions, and lifecycle warnings.
• Runtime behavior: filesystem layout, locks, GPU leases, process supervision, fingerprints, and resume.
• MCP setup: installed-package agent onboarding and client-owned setup.
• MCP operator reference: catalog fields, tools, authorization, run state, and recovery.
• Toy experiment and MCP catalog: a checkout-local CLI example backed by the packaged fake trainer, plus an MCP catalog whose detached-run state and experiment outputs use absolute scratch paths under /tmp.
• Tiny Decoder Enwik8 example: real-trainer integration.
• Durability invariants: the ordering rules the ledger, publication, and recovery paths must hold, and the test that owns each one.
• Development: source checkout, contributor setup, and quality gates.

License

MIT. See LICENSE.