beaker

Beaker SDK and CLI package.

beaker gives developers the tools to define optimization specs, validate them locally, upload datasets, configure hosted run environments, launch optimization runs, and inspect results from code or the CLI.

Customer specs import contract types directly from beaker:

from beaker import Case, CaseDataLoader, CaseResult, CaseScore, Spec, spec

Hosted model-selection rollouts also expose a generic OpenAI-compatible target:

from beaker import RolloutContext, inference_target


def build_eval_client(runtime: RolloutContext):
    target = inference_target(runtime)
    return YourFrameworkClient(
        base_url=target.base_url,
        api_key=target.api_key,
        model=target.model,
    )

Call inference_target only in the evaluation path when runtime.model is set. Production application calls should retain the application's existing model and client defaults. The structural beaker run smoke check does not invoke this path.

CLI run lifecycle

The CLI is intentionally noninteractive: choose the remote GitHub branch, dataset, run type, and models before invoking it. --ref resolves a remote GitHub ref, so unpushed local changes are not included.

# Discover launch inputs.
beaker github branches --repo owner/repository
beaker model list --available-only

# Ordinary prompt optimization.
beaker run trigger --ref feature/prompts --dataset invoices@production

# Optimize the production system without an initial benchmark.
beaker run trigger --ref feature/prompts --dataset invoices \
  --execution-mode optimize_only

# Benchmark and optimize selected models.
beaker run trigger --ref feature/prompts --dataset invoices \
  --optimization-model openai:gpt-4o \
  --optimization-model anthropic:claude-sonnet-4-5 \
  --execution-mode benchmark_and_optimize --benchmark-split TEST

# Discover, inspect, and cancel runs.
beaker run list
beaker run status RUN_ID
beaker run status RUN_ID --watch
beaker run cancel RUN_ID

Lifecycle commands support stable JSON for automation:

beaker model list --json
beaker github branches --repo owner/repository --json
beaker run trigger --ref feature/prompts --dataset invoices --json
beaker run list --json
beaker run status RUN_ID --json
beaker run cancel RUN_ID --json