agent_s4

The Agent S4 package, from the paper Evolving Agent Interfaces. It evolves a shadow interface: a small Python tool library that a computer-use agent imports to operate a desktop. Each step, a Builder agent edits an interface on the host. A task agent then runs benchmark tasks on fresh AWS VMs with that interface installed, and its score becomes the variant's fitness.

This file describes the code. For setup, credentials and the launch scripts behind the published runs, see the workspace READMEs in the Agent-S repository:

• osworld_setup/s4/README.md: OSWorld-V2, which is the default benchmark.
• scienceboard_setup/s4/README.md: ScienceBoard.

Install

agent-s4 is its own package; installing gui-agents does not include it.

The package contains only the code. A run also reads setup files from the Agent-S repository: the Builder prompts, the seed interface, the task splits and the pinned Prime Agent runtime, all under osworld_setup/s4/. So you need a clone of the repository either way. Install the package from the clone:

git clone https://github.com/simular-ai/Agent-S.git
cd Agent-S
pip install -e ./agent_s4

or install it from PyPI and point it at the clone:

pip install agent-s4
export SHADOW_EVOL_SHARED_ROOT=/path/to/Agent-S/osworld_setup/s4

Without those files, python -m agent_s4.cli run stops and says what to set. The benchmark itself (the OSWorld-V2 checkout, AWS and model credentials) is set up as described in the workspace READMEs above.

Entry points

run defaults to --benchmark osworld_v2, which uses the splits pinned in config/osworld_v2_release_pin.json: osworld_v2_train_50.json, osworld_v2_val_25.json and osworld_v2_test.json as the held-out set. No train or val splits ship for --benchmark osworld (the original OSWorld) or --benchmark scienceboard, so both require --train-set and --val-set.

How a run works

1. Bootstrap (orchestrator.run):

• If Prime Agent is used, checks its pinned runtime. For OSWorld-V2 it also runs the workspace preflight and checks the configured splits against the release pin.
• Makes one clone of the benchmark's Evol checkout per island, named <template>_<island> (for example OSWorldV2_Evol_<island>).
• Copies the seed interface into the clone's shadow_interface/ directory, which has its own git repository, and records it as the seed variant.
• Evaluates the seed on every island in parallel, writes the resolved config to runs/<timestamp>/config.json, and hands over to the driver.

2. Driver loop (driver.Driver):

• Keeps one global step counter.
• Each step, the scheduler (config/scheduler.py) picks one operator, which runs on every island in parallel. The step ends only when all islands have finished.
• Every dispatch is logged in the database, so a step interrupted by a crash is resumed rather than run twice.
• Validation reevaluation runs every --reevaluation-interval steps.
• The loop stops at --max-steps or --max-cost.

3. Operators (operators/): each step runs one of these.

A mutation runs these steps:

1. A tournament picks the parent (tournament.py).
2. The parent is checked out, and its task trajectories are staged for the Builder to read.
3. Optionally, a scratch VM is started (scratch_vm.py) so the Builder can try things in a live app.
4. The Builder harness edits shadow_interface/. For OSWorld-V2 and ScienceBoard it runs inside a Landlock sandbox, because the Base checkout next to the clone holds the benchmark's gated evaluators.
5. Edits outside the allowed scope are reverted, and the child is tagged evol/<island>/if-NNN.
6. The evaluator harness runs the child on the train and val splits. The per-task results, scores and costs go into the database.

Module map

Where files go

paths.py sets two roots:

• Shared root (SHADOW_EVOL_SHARED_ROOT; defaults to osworld_setup/s4/): the vendored Prime runtime, vm_assets/, the seed interfaces and the generic Builder prompts.
• Project root (SHADOW_EVOL_PROJECT_ROOT; defaults to the shared root): the run workspace. It holds the benchmark's prompts, splits, data, release pin, checkouts and every run artefact. The ScienceBoard launchers set it to scienceboard_setup/s4/.

Run artefacts are named per benchmark (and per SHADOW_EVOL_RUN_NAMESPACE) by benchmark_spec.ns(). For example, OSWorld-V2 uses shadow_evolution_osworld_v2_<ns>.db, eval_cache_osworld_v2_<ns>/ and builder_logs_osworld_v2_<ns>/.

Extending

Add an agent harness:

• Subclass BuilderHarness (harness/base.py) in harness/<name>/builder.py.
• Put the VM install and task-agent command in harness/<name>/evaluator.py.
• Register the key in both BUILDER_HARNESSES and EVALUATOR_HARNESSES.
• Teach UbuntuVMEvaluator and eval/osworld.py to dispatch on the key. Both currently accept only prime_agent and opencode.
• Pin and verify the upstream release in a source.py, and record it in a README, as the existing two harnesses do.

Add a benchmark:

• Add an entry to benchmark_spec.BENCHMARKS, and to _DEFAULT_SPLITS in config/experiment.py if it ships default splits.
• Write an eval worker that plugs into eval.osworld.main(impl=...), as eval/scienceboard_prime.py does.
• The benchmark-specific branches in harness/ubuntu_vm.py and orchestrator.py (search for config.benchmark ==) cover how the base checkout is chosen, which prompt addenda are added and how splits are checked.