benchmax

benchmax envs is where you define datasets, how to execute the rollout, and scoring each rollout.

for installation and project setup, start with the main readme. working environments live in examples/.

choose an environment

all benchmax environments implement the same dataset and rollout contracts. choose the adapter based on who owns the agent loop:

most custom environments should extend BaseEnv. most Harbor users configure HarborEnv directly rather than subclassing it.

sft datasets

supervised finetuning does not use an environment. construct a validated benchmax.sft.SftDataset from chat-format rows and upload it through the castform package; validation is strict and all-or-nothing.

architecture

an environment defines its dataset and how a group of rollouts runs against each example.

Environment
├── create_dataset(split, base_dir, max_examples)
│   └── Dataset
│       └── Example(id, payload)
└── run_group(requests)
    ├── run_rollout(request) × group_size → RolloutAttempt × group_size
    ├── adapter-specific scoring
    ├── optional group-relative scoring
    └── RolloutOutcome(rewards, termination_reason, error)

every environment follows this shape. the environment decides what an example contains, how each attempt runs, which tools are available, and how the result is scored.

datasets

create_dataset receives a train or eval split and returns a fixed, ordered Dataset of Example objects.

each example contains:

• a stable id used to identify the datapoint across runs;
• an environment-owned payload consumed by its rollout implementation.

the optional max_examples argument limits how many examples are returned. when the data source supports it, the environment should stop loading once it reaches that limit.

JsonlDataset, Harbor datasets, and custom datasets all produce the same fundamental Dataset type. the trainer and validation flow do not depend on the source file format.

tools

BaseEnv exposes OpenAI-compatible function tools through list_tools and executes them through run_tool. an environment can provide no tools, one tool, or a collection of stateful tools.

with HarborEnv, the Harbor agent and harness define the available tools and how they interact with the sandbox. benchmax does not convert Harbor tools into BaseEnv tools.

execution and scoring

run_group receives multiple rollout requests for the same example, runs them concurrently, waits for all siblings, and returns one RolloutOutcome for each request.

successful scoring hooks return their named reward components. operational failures return no rewards and do not cancel successful siblings; the trainer treats absent components as zero. partial attempts that reach a context, output, turn, or tool limit can still be scored.

• BaseEnv runs the model and tool loop, then passes the transcript and example payload to compute_reward. compute_group_rewards can score the completed sibling group.
• HarborEnv runs the configured Harbor agent and sandbox, then preserves its verifier or RewardKit reward components.

helpers

benchmax.rewards provides deterministic text helpers, model judges, rubrics, ranking, adaptive rubrics, and diversity scoring for BaseEnv and direct Environment implementations. see the rewards guide.

Harbor environments normally use their harness verifier and RewardKit instead of benchmax reward helpers.

bundling

a bundle contains the environment class, its constructor arguments, the project-local source it needs, and its declared remote dependencies.

environment class + constructor arguments
                 + local source
                 + dependency metadata
                           │
                           ▼
                    portable bundle

benchmax creates the portable artifact so the same environment can be loaded outside the author's checkout. castform handles uploading the bundle, validating it remotely, and using it for training.

Remote runtimes install pip_dependencies without enabling prereleases globally. If a dependency graph needs a prerelease, list that package explicitly even when it would normally be transitive. For example, use pip_dependencies=["parent-package==1.0.0", "transitive-package==2.0.0rc1"].

further reading

• base environment guide
• harbor environment guide
• reward helpers
• examples
• development instructions

apache 2.0 © 2026 CGFT Inc.