marble-v1

Place pieces so a falling marble lands in a ring. One reply, no tools, no simulator to poke at — the whole task is predicting a chain of bounces before any of them happen.

uv run eval @ eval.toml -m <model> \
  --client.base-url <provider> --client.api-key-var <VAR>

uv run eval @ eval.toml -m <model> --env.taskset.pieces 3   # harder

What makes it different

The reward cannot be argued with. There is no judge model and no rubric. A fixed-step physics integrator runs the layout and reports whether the marble went through the ring. Reward hacking is not defended against here; it is impossible.

Solvable by construction. Levels are built forwards along the marble's real path, so a working layout is known before the task is handed out. A model that fails failed at something possible — which is not true of levels that are designed and then hoped over.

Infinite, and difficulty is one number. pieces is the knob. Levels stream from a seed; there is no dataset to exhaust.

Cheap to run. No browser, no VM, no judge calls. A rollout is one model call plus a few milliseconds of arithmetic — the difference between a training run being affordable and not.

The task

A marble rolls off a fixed ramp with a known position and velocity and falls under gravity. The model is given the ring's position and mouth width, an inventory of pieces it must use exactly, each piece's size, bounciness and surface grip, and the placement grid. It replies with a JSON layout.

Five pieces behave differently: a spring fires the marble along its own axis, so rotating one aims it; a ladder is a concave arc that cradles; drum, block and bell are rounded boxes with different bounce. All placements snap to a 20-unit grid and 7.5-degree rotation steps, so the action space is integral and an answer is either legal or it is not.

Rewards

The two diagnostics carry no weight on purpose. closeness is kept out of the training signal because a model paid for near misses learns to aim near the ring rather than through it. answered_legally exists because "could not format an answer" and "got the physics wrong" are very different failures that otherwise produce the same zero.

Difficulty

Two knobs, and you need both. pieces alone is far too coarse — measured, it goes 50% → 0% between one piece and two, and a cliff gives a training run nothing to climb. ring_half_width is the fine one: a wider mouth converts near misses into hits without changing what the model has to reason about.

Measured on google/gemini-2.5-pro, 10 tasks each, null harness:

The mouth knob moves the solve rate 0 → 10 → 30% at a fixed piece count, so any model can be put in a band where it solves neither everything nor nothing.

answered_legally is 100% everywhere: no model in these runs failed to produce a legal layout. Every zero above is a physics failure, not a formatting one.

Baselines

Measured on generated levels, no model involved (calib.py):

The first column is the floor a model has to beat to be doing anything — 50% at one piece is a 13x lift over guessing. The second says the solution is a basin rather than a needle: the task rewards approximately right physics, not exact recall.

Widening the mouth does not hand the win to chance. At two pieces, taking the mouth from 118 to 420 moves the random floor only 1.8% → 6.4%, while it moves the model 0% → 30%.

What a rollout costs

The environment's own overhead is a few milliseconds of arithmetic — no browser, no VM, no judge model. The model is what costs: a reasoning model produces long chains on these problems, and gemini-2.5-pro averaged about $0.16 per rollout in the runs above. Calibrate with a cheaper model first.

Provenance

The simulator is a port of the one in a shipped iOS game, where determinism was the feature rather than a detail: a player who nudges one piece and re-runs has learned nothing if the same layout can give two answers. It is a fixed-step integrator with analytic contacts, not a physics engine.

The port is checked against the original: on a five-piece reference layout the marble's finishing position matches exactly, and intermediate crossings agree to within one unit. Bit-identity across languages is explicitly not a goal — sin and cos may differ in the last place — so reference solutions are generated by this simulator rather than imported from it.

Files