beaker

Beaker SDK and CLI package.

beaker gives developers the tools to define optimization specs, validate them locally, upload datasets, configure hosted run environments, launch optimization runs, and inspect results from code or the CLI.

Customer specs import contract types directly from beaker:

from beaker import Case, CaseDataLoader, CaseResult, CaseScore, Spec, spec

Hosted model-selection rollouts also expose a generic OpenAI-compatible target:

from beaker import RolloutContext, inference_target


def build_eval_client(runtime: RolloutContext):
    target = inference_target(runtime)
    return YourFrameworkClient(
        base_url=target.base_url,
        api_key=target.api_key,
        model=target.model,
    )

Call inference_target only in the evaluation path when runtime.model is set. Production application calls should retain the application's existing model and client defaults. The structural beaker run smoke check does not invoke this path.

CLI run lifecycle

Every command, the endpoint it calls, and who can run it are documented in docs/cli-reference.md.

The CLI is intentionally noninteractive: choose the remote GitHub branch, dataset, and optional comparison models before invoking it. Runs use the repository's GitHub default branch when --ref is omitted. --ref resolves a remote GitHub ref, so unpushed local changes are not included.

Datasets are train + test. The TRAIN split drives optimization; TEST is the held-out split scored against the baseline. beaker dataset upload requires both split counts.

# Upload a dataset with its split counts.
beaker dataset upload ./data --name invoices --split train=400 --split test=50

# Discover launch inputs.
beaker github branches --repo owner/repository
beaker model list --available-only

# Agent optimization of the production system.
beaker run trigger --ref feature/prompts --dataset invoices@production

# Agent optimization comparing selected models.
beaker run trigger --ref feature/prompts --dataset invoices \
  --optimization-model openai:gpt-4o \
  --optimization-model anthropic:claude-sonnet-4-5 \
  --benchmark-split TEST

# Discover, inspect, and cancel runs.
beaker run list
beaker run status RUN_ID
beaker run status RUN_ID --watch
beaker run cancel RUN_ID

Lifecycle commands support stable JSON for automation:

beaker model list --json
beaker github branches --repo owner/repository --json
beaker run trigger --ref feature/prompts --dataset invoices --json
beaker run list --json
beaker run status RUN_ID --json
beaker run cancel RUN_ID --json