beaker

Beaker SDK 0.6.4 defines a customer Integration, validates it locally, uploads datasets and operates hosted optimization runs. Import contract types from beaker and declare a module-level integration = Integration(...) value.

from beaker import (
    Integration,
    Case,
    CaseResult,
    CaseScore,
    RepositoryRunSetup,
    RolloutRuntime,
    SetupRuntime,
    repository,
)

integration = Integration(
    targets=repository(),
    run_setup=MySetup,
    run_case=run_case,
    score_case=score_case,
)

MySetup subclasses RepositoryRunSetup, declares a Pydantic-compatible row_model, and implements async-generator load_cases(row, *, runtime). It yields Case(id=..., input=..., expected=...) values with unique IDs. Beaker constructs one setup instance per attempt and enters/closes its prepare_run() async context manager.

run_case(*, case_input, runtime) returns CaseResult(output=...). Its JSON application result is what the scorer reads; telemetry belongs in runtime.trace. score_case(*, case, result, case_files_dir) returns CaseScore with the agreed objective, stable field metrics and optional per-case checks.

For document optimization, use documents(groups=(...)) and DocumentRunSetup. Setup provides TargetDocument seeds. An optional open_candidate() creates a runtime from candidate files; document results are conditional ChangeSet operations. The customer decides when to apply them.

Configuration and validation

beaker init preserves existing configuration. For a new setup, it creates .beaker/beaker_integration.py and:

default_integration: invoices
integrations:
  invoices:
    entrypoint: beaker_integration:integration
    source_dir: .
    package_import_root: .beaker
    required_env: []

You can add targets to an Integration entry so playbook generation can start while its image builds. For example, a repository Integration can declare:

    targets:
      kind: repository
      paths: ["src"]

The declaration must match Integration.targets. At run startup, Beaker imports the Integration with its runtime environment and secrets, then compares its targets with the declaration saved for that build. A mismatch stops the run before prepare_run() or baseline evaluation and reports both target declarations. The image build does not import the Integration for this check. Playbook generation can still start during the build; a mismatch can therefore waste generation work, but the run cannot use the mismatched declaration for optimization. The configured entrypoint selects the module and attribute; the Integration name does not change how the module is resolved.

source_dir is relative to the Git root; package_import_root is inside it. Use --integration-id when several entries exist. Config selection is explicit ID, configured default, or the sole entry; ambiguous selection fails. To register an Integration elsewhere, pass beaker init --integration-id <id> --target <path.py-or-module:attribute>. Init does not scan the repository for Integrations.

beaker agent setup "<Agent Name>" --integration-id invoices records the hosted agent key at integrations.invoices.agent_key. Agent selection uses --agent / --agent-key, then BEAKER_AGENT_KEY, then the selected integration's key. Top-level agent_key and project_key fields are no longer used.

beaker onboarding status --integration-id invoices
beaker agent env check --integration-id invoices
beaker run smoke --strict --integration-id invoices --dataset invoices@revision
beaker run trigger --integration-id invoices --dataset invoices@revision
beaker run status RUN_ID --watch
beaker run pull RUN_ID

Smoke validates all rows before entering setup, then loads cases, validates input files, and opens a document seed candidate if applicable. It closes all resources without invoking the agent or scorer. Setup hooks may use external services. Init verification only imports and validates the Integration.

Models

Both repository and document Integrations support explicitly selected comparison models using repeated --optimization-model provider:model flags. When runtime.model is present, inference_target(runtime) provides the hosted gateway configuration. Otherwise preserve the application's production defaults. Set an optional fixed judge with launch scorer_model and call scoring_inference_target() inside scoring.

Read the CLI reference for commands.

The SDK and skills are released together with matching versions. Deploy the compatible hosted runtime before publishing the SDK and skills.