beaker

Website · Documentation

Beaker SDK 0.6.5 defines a customer Integration, validates it locally, uploads datasets and operates hosted optimization runs. Import contract types from beaker and declare a module-level integration = Integration(...) value.

from beaker import (
    Integration,
    Case,
    CaseResult,
    CaseScore,
    RolloutRuntime,
    RunSetup,
    SetupRuntime,
    repository,
)

integration = Integration(
    targets=repository(),
    run_setup=MySetup,
    run_case=run_case,
    score_case=score_case,
)

MySetup subclasses RunSetup, declares a Pydantic-compatible row_model, and implements async-generator load_cases(row, *, runtime). It yields Case(id=..., input=..., expected=...) values with unique IDs. Beaker constructs one setup instance per attempt and enters/closes its prepare_run() async context manager, which yields one result per declared target kind; the default yields {"repository": RepositoryRunSetupResult()}.

run_case(*, case_input, runtime) returns CaseResult(output=...). Its JSON application result is what the scorer reads; telemetry belongs in runtime.trace. score_case(*, case, result, case_files_dir) returns CaseScore with the agreed objective, stable field metrics and optional per-case checks.

For documents, declare documents(groups=(...)) and override prepare_run(). Declare it alone to optimize only documents, or beside the repository to let one candidate change code and documents together: targets=(repository(...), documents(groups=(...))). Declare one documents() target per document store (two S3 buckets of wikis are two targets); group names must be unique across them, and there is at most one repository(). prepare_run() yields one result per target kind, such as {"repository": RepositoryRunSetupResult(), "documents": DocumentRunSetupResult(...)}: a single documents result holds the TargetDocument seeds of every document target, and each document's group says which target it belongs to. run_case reads the candidate's documents from runtime.targets_dir. To build state every case of a candidate shares, such as an index of its documents, pass Integration(open_candidate=...), an async context manager taking (*, targets_dir, scratch_dir): it is entered once per rollout process before any case, its value reaches run_case as runtime.candidate, and it is exited after the last case. Each document target's result is its own DocumentChangeSet of conditional operations, naming its groups; the customer decides when to apply them.

Configuration and validation

beaker init preserves existing configuration. For a new setup, it creates .beaker/beaker_integration.py and:

default_integration: invoices
integrations:
  invoices:
    entrypoint: beaker_integration:integration
    source_dir: .
    package_import_root: .beaker
    required_env: []

You can add targets to an Integration entry so playbook generation can start while its image builds. For example, a repository Integration can declare:

    targets:
      kind: repository
      paths: ["src"]

The declaration must match Integration.targets. At run startup, Beaker imports the Integration with its runtime environment and secrets, then compares its targets with the declaration saved for that build. A mismatch stops the run before prepare_run() or baseline evaluation and reports both target declarations. The image build does not import the Integration for this check. Playbook generation can still start during the build; a mismatch can therefore waste generation work, but the run cannot use the mismatched declaration for optimization. The configured entrypoint selects the module and attribute; the Integration name does not change how the module is resolved.

source_dir is relative to the Git root; package_import_root is inside it. Use --integration-id when several entries exist. Config selection is explicit ID, configured default, or the sole entry; ambiguous selection fails. To register an Integration elsewhere, pass beaker init --integration-id <id> --target <path.py-or-module:attribute>. Init does not scan the repository for Integrations.

beaker agent setup "<Agent Name>" --integration-id invoices records the hosted agent key at integrations.invoices.agent_key. Agent selection uses --agent / --agent-key, then BEAKER_AGENT_KEY, then the selected integration's key. Top-level agent_key and project_key fields are no longer used.

beaker onboarding status --integration-id invoices
beaker agent env check --integration-id invoices
beaker run smoke --strict --integration-id invoices --dataset invoices@revision
beaker run trigger --integration-id invoices --dataset invoices@revision
beaker run status RUN_ID --watch
beaker run pull RUN_ID

Smoke validates all rows before entering setup, then loads cases, validates input files, and opens a document seed candidate if applicable. It closes all resources without invoking the agent or scorer. Setup hooks may use external services. Init verification only imports and validates the Integration.

Models

Integrations support explicitly selected comparison models using repeated --optimization-model provider:model flags. When runtime.model is present, inference_target(runtime) provides the hosted gateway configuration. Otherwise preserve the application's production defaults. If scoring requires a hosted LLM judge, declare requires_scorer_model: true and scorer_model on the selected integration in .beaker/beaker.yaml:

integrations:
  my-agent:
    # ...existing integration settings...
    requires_scorer_model: true
    scorer_model: openrouter:deepseek/deepseek-v4-flash

Call scoring_inference_target() inside scoring. Judge selection is owned by the integration, not launch settings or config_defaults; CLI/API overrides are rejected. Missing or invalid required judges fail before creating a build or run. Provider credentials are checked before launch. Integrations without LLM scoring can still launch without a judge.

Read the CLI reference for commands.

The SDK and skills are released together with matching versions. Deploy the compatible hosted runtime before publishing the SDK and skills.