ReAct-Manip v0.3

Robot asset update: The active embodiments now use official PiPER-X arms: piperx, dual_piperx, and mobile_dual_piperx (custom Tracer assembly). Build with python scripts/build_agilex_assets.py; pinned upstream revisions, URLs, and SHA-256 digests are recorded in scripts/piperx_assets.lock.json. The PiPER-X candidate bank contains regenerated initial states and main/wrist camera observations. Old ALOHA selections remain incompatible; draw from the new bank below.

Optional HarnessVLA and four RPent native-embodiment integrations live under integrations/harness_vla/ and integrations/rpent_embodiments/. They do not change the task defaults or the maintained PiPER-X evaluation bank.

简体中文

A robotics evaluation runtime for external policies, with 16 tasks, a fixed candidate bank, and one evaluation entry point. Optional features include a policy-view window, video recording, and keyboard control through Pygame. Training and reference solving policies are outside this release. Robot asset construction and rebuilding existing resolved instance banks are supported by scripts.

Full evaluation scope: A complete ReAct-Manip evaluation must run all 16 tasks in the main bank, all three Task Instances for every task, and all 15 Episodes for every instance (10 practice + 5 exam): 48 evaluation groups and 720 Episodes in total. A single task, task subset, single instance, shortened run, smoke check, aborted run, or otherwise incomplete run is for debugging or partial analysis only and must not be reported as a complete Benchmark result.

Background: why we built this Benchmark

Physical manipulation is sensitive to changes in the robot, workspace, object geometry, mass, friction, contact dynamics, and visual conditions. A policy that succeeds in one setting may fail after a small change. Reliable generalization therefore involves more than recognizing a task or producing an initial action: the robot must observe the consequences of its actions and respond to the environment.

ReAct-Manip studies this process of acting, observing, learning, and adapting. Zero-shot performance is a useful baseline, but contact-rich tasks also require feedback, repeated attempts, and recovery. We build tasks that make these capabilities measurable through reproducible physical interactions, including cases where an initially plausible strategy does not work.

The practice/exam protocol gives this motivation a concrete experimental setting. A policy receives 10 practice episodes followed by 5 exam episodes on the same Task Instance. It can retain experience across those episodes and continue adapting during exams; a fresh policy is created for the next Task Instance. The primary score measures exam performance. Practice trajectories, failures, and recovery behavior help explain how that performance was reached. The runtime provides the evaluation setting; the external policy determines whether and how to learn.

Across those 15 Episodes, all four closed-loop parameter-adaptation tasks continuously change task-critical conditions. The policy does not face one fixed geometry and dynamics configuration: it must infer the current condition from action consequences while carrying useful probing and correction methods forward from practice. A complete evaluation runs every candidate and every saved parameter sequence; selecting only favorable Episodes cannot produce a complete score.

Use cases

• In-context adaptation: use task language and interaction history to infer properties of the current instance and improve subsequent actions.
• Continual adaptation within a Task Instance: compare performance across practice and exam episodes while retaining policy state. The public protocol resets policy state between Task Instances, so it does not by itself measure lifelong learning across tasks.
• Generalization across scenes and tasks: study sensitivity to the geometry, physical parameters, object choices, and initial conditions that each task actually samples. Randomization is task-specific, not a promise that every parameter varies in every task.
• Embodiment research: develop policies for the single-arm, fixed dual-arm, and mobile dual-arm PiPER-X configurations. Each released task declares its supported robot; changing that embodiment requires compatible assets, task validation, and regenerated instances rather than a bank-mode CLI override.
• Closed-loop behavior analysis: inspect actions, observations, contacts, task events, errors, and recovery using the feedback available for that task, alongside final scores.
• Training and annotation pipelines: collect simulator geometry truth for internal supervision, diagnostics, and Oracle comparisons, including a separate ../CL_Coach checkout. Training algorithms and CL_Coach itself are outside this release.
• Simulation-to-real preparation: investigate collision, reachability, and contact sensitivity before hardware experiments. Simulator performance alone does not establish hardware calibration or real-world transfer.

Benchmark design principles

1. Evaluate adaptation through interaction. An unsuccessful attempt is useful evidence. Retries, recovery, and the time available for them follow the task's rules; success is not assumed for an Oracle or a reference strategy.
2. Use physical execution as the source of truth. Runtime actions must flow through the task's controller, actuators, and MuJoCo contact dynamics. Initialization and explicit replay may restore saved state; writing poses or velocities during an attempt to manufacture success is not a valid solution.
3. Separate rules, instances, and episodes. TaskSpec and the instance contract define the task; Task Instances resolve the parameters they own; Episode Instances specify their permitted initial-state or instruction variation. Profiles describe sampling distributions. Ownership is declared per task so that randomization remains controlled and reproducible.
4. Separate public observations from privileged information. Proprioception, policy images, language, and declared feedback form the public interface. Simulator geometry, hidden assignments, and object locations belong to a separate internal channel. Comparisons must state which information a policy received.
5. Keep time and scoring explicit. Physics advances synchronously with control steps. Task-defined completion, timeout, and score rules govern each episode; exam scores are aggregated by the protocol. Diagnostics and wall-clock inference latency help interpret results but do not replace the declared score.
6. Make comparisons reproducible. Preserve task and episode identities, resolved parameters, source/model/controller digests, dependency versions, and the selected policy track. Changed task semantics or physics require compatible instances; old and new results should not be silently mixed.
7. Keep extensions declarative and evidence scoped. Register tasks and geometry adapters through the existing interfaces. Distinguish successful loading, exact replay, a physical target-state witness, and a complete robot solution: none automatically proves the next.

Cross-Episode parameter changes in closed-loop tasks

The main bank stores explicit Episode-level variations for all four closed-loop parameter-adaptation tasks: three Task Instances per task and 15 Episodes per instance, for 180 parameter-varying Episodes. Parameters are resolved before an Episode starts and remain fixed within it. Resetting the same Episode replays the exact saved condition instead of resampling it.

Task	Condition changed between Episodes	Domain
leveling_board1	3D dimensions of all six shims; identity and thin-axis orientation of the three thin shims	Ordinary dimensions 6–14 cm; thin dimensions 3, 4, and 5 cm
conveyor_grasp	Conveyor speed and the size of every target object	0.08–0.20 m/s; target scale 0.80–1.15×
box_insertion	Sliding friction of the left/right contact regions and total block mass	Friction 0.6–1.4× baseline, clipped to [0.01, 0.6]; mass 0.7–1.3× baseline
car_drive2	Vehicle nominal-speed gain associated with throttle displacement	0.90–1.15 m/s

Continuous variables use five-stratum coverage. Practice Episodes 1–5, practice Episodes 6–10, and exam Episodes 1–5 each cover the same full domain in an independently shuffled order. Practice and exam therefore share a parameter domain but differ in exact values and ordering. A public Policy is told which properties may change, but never receives the current hidden samples; it must estimate and respond through vision, contact, motion consequences, and task feedback.

The variations are applied before MuJoCo model compilation and initial-state capture, then stored with the resolved runtime, model identity, and initial integration state in each Episode Instance. phone_lean also changes phone/support initial positions between Episodes, but support identities, sizes, masses, friction, and gravity belong to the Task Instance and stay fixed across its 15 Episodes; it is therefore reported separately from these four parameter-adaptation tasks.

Geometry ground-truth Feature interface (for ../CL_Coach)

Geometry Features are an internal, read-only channel for data collection, annotation, diagnostics, and explicitly privileged Oracles. They are obtained through GeometryFeatureAdapter, separately from the public observations sent to a policy. ../CL_Coach refers to a separate consumer checkout; it is not a bundled dependency or an automatically started service.

Adapters read the compiled MuJoCo model, simulation data, and task-owned read-only state. Extraction must not advance physics, change qpos/qvel, or alter scoring and dwell timers. Body poses use world coordinates, meters, and MuJoCo (w, x, y, z) quaternions. Velocity fields use SI units; matrices, local coordinates, and fields explicitly suffixed _deg retain their task-defined conventions. Check field definitions instead of treating every numeric array as a pose.

The stable top-level schema is:

{
  "schema_version": "geometry.v1",
  "entities": {"<entity name>": ...},
  "relations": {"<relation name>": ...},
  "task_state": {...},
  "validity_mask": {...}
}

Entity contents and array sizes vary by task; some entries are lists of objects. The validity mask describes the adapter's reported entities, not a universal sensor-quality certificate. Geometry diagnostic fields are not the scoring authority: use the task's declared final info[score_key] and terminal feedback for evaluation.

Collection-side interface

Keep the returned environment wrapper for reset() and step(), and unwrap it only to select the task's geometry adapter. This minimal internal collection example uses the button task and one zero-action transition; it is not a solving policy or a scored evaluation:

import numpy as np
from react_manip.builtin import discover_task_definitions
from react_manip.core import GeometryFeatureAdapter, normalize_reset, normalize_step
from react_manip.core.compatibility import resolve_task_robot

spec, factory = next(
    (spec, factory) for spec, factory in discover_task_definitions()
    if spec.task_id == "button_indicator_matching"
)
robot_id = resolve_task_robot(spec)
env = factory(seed=0, robot_id=robot_id,
              has_offscreen_renderer=False, use_camera_obs=False)
try:
    observation, info = normalize_reset(env.reset())
    task_env = getattr(env, "env", env)
    adapter = GeometryFeatureAdapter.for_task(task_env)
    before = adapter.extract()
    transition = normalize_step(env.step(np.zeros(env.action_dim)))
    after = adapter.extract()
    record = {
        "task_id": spec.task_id,
        "robot_id": robot_id,
        "scenario_id": getattr(getattr(env, "scenario", None), "scenario_id", None),
        "simulation_time_s": float(task_env.sim.data.time),
        "features": after,
        "reward": transition.reward,
        "terminated": transition.terminated,
        "truncated": transition.truncated,
    }
finally:
    env.close()

Extract a fresh snapshot after each step and align it with that transition's action, public observation, language, feedback, and simulation timestamp. A training dataset should additionally record Task/Episode IDs, scenario and runtime digests, phase, step index, schema version, and camera IDs when images are enabled. NumPy arrays require an array-capable storage format or explicit conversion before JSON serialization. The example disables RGB; collection requiring images should enable the documented camera interface on the same environment.

Keep privileged Features in a separate dataset field or stream. Do not use robosuite's concatenated object-state as a cross-task schema, and do not pass an unfiltered raw observation directly to a public policy. The runner applies policy_observation and policy_info; declared legacy privileged keys are filtered, and geometry truth is attached only for an explicit internal Oracle opt-in. Public candidate-bank evaluation rejects requires_geometry_truth=True. --trace-dir records the public evaluation stream; it is not a substitute for this internal geometry collector. The current contracts remain task-specific, so a data pipeline must audit which additional state fields are public.

Available adapter entry points

These are the adapters currently implemented in this release, not a claim that all 16 tasks expose geometry Features. Most live in react_manip/core/geometry.py; the drawer-search adapter registers from its task module when that environment is imported.

Task	Adapter	Feature contents
box_insertion	BoxInsertionGeometry	Block and end-effector geometry, slot bounds, clearance, contact force, and insertion state
laptop_into_drawer	LaptopIntoDrawerGeometry	Laptop, drawer, handle, and end-effector geometry; containment bounds and relative distances
find_object_in_drawer2	FindObjectInDrawer2Geometry	Object poses and drawer membership, drawer slides, destination-box bounds, current target, and placement state
conveyor_grasp	ConveyorGraspGeometry	Object poses, velocities, categories, bins, examples, assignments, and sorting state
phone_lean	PhoneLeanGeometry	Phone, selected supports, and end-effector poses; relative positions and diagnostic distance/height fields
pitch_pot	PitchPotGeometry	Projectile pose/velocity, pot and tray geometry, relative position, and hit/attempt state
button_indicator_matching	ButtonIndicatorMatchingGeometry	Buttons, indicator labels and states, hidden wiring, current target, raw score, and events

For a new task, subclass GeometryFeatureAdapter, register its task ID with @GeometryFeatureAdapter.register("my_task"), implement extract() using _result(...), and declare the field semantics and visibility in TaskSpec. Add reset/step checks for shapes, validity, and read-only behavior. Collection must report a missing adapter explicitly rather than inventing a generic geometry view. Existing registration does not require a task-specific branch in the runner or Pygame frontend.

Installation

The candidate bank requires Python 3.10, NumPy 1.26.4, MuJoCo 3.3.1, and robosuite 1.5.2. Install the released package and the asset-building dependencies in a Python 3.10 environment:

python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install "react-manip[assets]==0.3.0"
react-manip-build-assets

Use react-manip[all] to include visualization and HarnessVLA integration dependencies. For development from a source checkout, python -m pip install -e ".[all]" is supported.

Robot assets are not included in the wheel because the generated bundle is too large for a normal PyPI distribution. The builder downloads checksum-locked official source models and meshes from pinned upstream commits. A source checkout keeps assets under assets/robots/agilex/; a wheel install uses ~/.cache/react-manip/assets/robots/agilex/ by default. Override either location with AGILEX_ASSET_DIR. The first build requires network access and later builds reuse cached sources. Small scene assets, textures, task definitions, and the candidate bank are included in the wheel. Do not substitute other object assets: the bank validates the compiled model identity.

The runtime defaults to NUMBA_DISABLE_JIT=0 before importing simulation dependencies, matching the recorded bank. Do not switch JIT modes within one process. Optional IK or private-configuration warnings from robosuite do not imply missing dependencies for the controllers used here.

Visual evaluation requires a working OpenGL context. macOS uses its default backend. Headless Linux typically uses MUJOCO_GL=egl with a working EGL driver, or MUJOCO_GL=osmesa with OSMesa installed. --state-only-smoke checks the pipeline without images and produces no protocol score.

Evaluation

The main-bank command below performs the complete evaluation. It does not use --draw-seed to choose only one candidate per task: it runs all 16 tasks, all 48 Task Instances, and all 720 Episodes, including all 180 parameter-varying Episodes from the four closed-loop tasks. The aggregate protocol_score represents a complete Benchmark result only when every group finishes validly; if any group is missing or invalid, the score is null. --task, task-specific manifests, and --state-only-smoke are not complete evaluations.

When neither --manifest nor --task is specified, evaluation defaults to the shipped react_manip/suites/piperx_candidates_v1/bank.json, resolved relative to the repository rather than the current working directory. --draw-seed remains required. Use --candidate-bank <path> to select another bank; explicit manifest and task modes retain their existing behavior.

For a live evaluation window or video recording, see First-person cameras and visualization. Add --visualize to bank or manifest evaluation; add --video-dir to save videos.

python scripts/evaluate_all_tasks.py \
  --draw-seed 42 \
  --policy my_policy:build \
  --out-dir outputs/model_seed42_full

evaluate_all_tasks.py is the complete-evaluation entry point. It invokes the lower-level evaluate_policy.py and additionally verifies the task count, 48 evaluation groups, 720 completed Episodes, and the 10/5 protocol shape before writing results, logs, and summary.json under the output directory. Place my_policy.py in the repository root or on the Python module search path. Each task has three candidate Task Instances, and candidate-bank evaluation runs all three. The draw seed is retained for reproducible selection provenance but no longer drops any instance; do not pass candidates.json directly as an evaluation manifest. Omitting --policy selects the random policy for pipeline checks; it is not a model result.

Evaluations also have wall-clock safeguards: by default, each episode is limited to 30 minutes. The aggregate limit for all episodes of one task is 2 hours for conveyor_grasp, 4 hours for find_object_in_drawer2, and 1 hour for other tasks. Completed results are retained, but a timed-out task/evaluation is marked invalid. Override these limits with --episode-wall-timeout-sec and --task-wall-timeout-sec. The limits run in the shared evaluator layer, so they also apply when calling BenchmarkRunner or the manifest suite directly.

By default, the draw is saved beside the result as <result_stem>_selection/, containing a replayable manifest and all three instances and their episode records for every task. The same bank and --draw-seed produce the same manifest contents. Result files, selection directories, and trace directories must use new paths. Selection sidecars generated by the old one-instance-per-task protocol are rejected; draw a fresh selection after this change.

# Replay the exact selected instances.
python scripts/evaluate_policy.py \
  --manifest outputs/model_seed42_full/all_tasks_selection/manifest.json \
  --policy my_policy:build --out outputs/model_seed42_repeat.json

# State-only pipeline check, using the default random policy; no protocol_score.
python scripts/evaluate_all_tasks.py \
  --draw-seed 0 --state-only-smoke --out-dir outputs/smoke_full

Use --trace-dir outputs/new_trace to record public observations, actions, and transitions. In bank/manifest mode, the records own the robot, scene, horizon, control frequency, and episode counts; CLI overrides are rejected. For task-default debugging, use --task TASK --policy my_policy:build. Its protocol follows the task declaration and is separate from fixed-bank evaluation.

External policy interface

This minimal zero-action adapter implements the interface; it is not a solving policy:

# my_policy.py
import numpy as np

class ModelPolicy:
    policy_id = "my-model"

    def __init__(self):
        self.history = []

    def set_action_dim(self, action_dim):
        self.action_dim = action_dim

    def reset(self, *, seed=None, task_spec=None):
        # Independent --task episodes: initialize your model state here.
        pass

    def begin_episode(self, *, phase, episode_index, seed, task_spec):
        # phase is practice or exam; episode_index starts at zero per phase.
        # Clear episode-local caches while retaining cross-episode learning.
        self.phase = phase

    def act(self, observation, info):
        # observation["images"]: {camera_id: RGB uint8 H×W×3 array}
        # info.get("language_prompt"): current task instruction
        # Map model outputs to the robot's action layout.
        return np.zeros(self.action_dim, dtype=np.float32)

    def end_episode(self, **feedback):
        self.history.append(feedback)

def build():
    return ModelPolicy()

The factory must return a fresh policy. Fixed-bank evaluation creates one policy per Task Instance and reuses it for all 15 episodes. Learning state persists between practice and exam; online adaptation is allowed during exams. Each new task receives a fresh policy. Avoid leaking learned state across tasks through module globals.

Actions must be finite vectors of length action_dim, with all values in [-1, 1]. Invalid actions invalidate the group; they are not silently clipped. Each Task Instance selects its declared PiPER-X embodiment and recorded controller layout; a seven-dimensional single-arm policy cannot be applied unchanged to a dual-arm instance. Actual observation dictionaries and TaskSpecs define the image, instruction, and proprioceptive inputs. Instance files are not passed to the policy. Observation filtering follows task declarations; some tasks still expose task state or physical parameters. The public track rejects policies with requires_geometry_truth=True.

An instance/group score is the mean of its five exam scores; the candidate-bank protocol score is the mean across all 48 groups (equivalently, each task's 15 exam episodes receive equal weight). If any group is invalid, the protocol score is null and the CLI exits with a nonzero status. official_score remains null: this pilot release is not a certification of solvability for every task. Simulation advances synchronously; slow inference does not consume extra control steps.

First-person cameras and visualization

PiPER-X evaluation supplies all public views through observation["images"] as 640×480 RGB uint8 arrays:

Embodiment	Camera IDs in order
piperx	robot0_head_camera, robot0_wrist_right
dual_piperx / mobile_dual_piperx	robot0_head_camera, robot0_wrist_right, robot0_wrist_left

The single-arm main camera is fixed above and behind the arm base. Dual models retain their main mounts. Wrist cameras now use the local pose (-0.1, 0, -0.08) and 65° field of view on the rigid gripper frame; their optical axis targets (0, 0, 0.06) and local -X is image-up. This first fit trial lifts the sensor 10 cm in the local image-up direction to reduce gripper-root occlusion while preserving the TCP aiming point. They follow the wrist, not finger opening, and never track targets. All views render the same simulation state without advancing physics. These are simulation sensor configurations, not hardware calibration. Pygame and evaluation use the same camera poses and image orientation; the display transpose only adapts array width/height to pygame and is not a 90° image rotation. Wrist images still roll with the wrist during wrist roll, while the neutral view is upright; the gripper can occlude part of the initial view, so use the main view alongside it.

Visualization dependencies are included in requirements.txt; no second install command is needed.

# Evaluate the main bank with a live window and video recording.
python scripts/evaluate_all_tasks.py \
  --draw-seed 42 \
  --policy my_policy:build \
  --visualize \
  --video-dir outputs/visual_eval_42_videos \
  --out-dir outputs/visual_eval_42

Run from the repository root in your configured Python environment. Replace my_policy:build with your policy entry point. Use new result, selection and video paths. --visualize requires a desktop display and a working OpenGL context; do not set MUJOCO_GL=disable. On a headless machine, omit --visualize and use --video-dir with a working offscreen rendering backend.

# Task-default debugging.
python scripts/evaluate_policy.py \
  --task box_insertion --policy my_policy:build \
  --visualize --video-dir outputs/new_camera_videos \
  --out outputs/new_camera_result.json

The task selects its configured robot. Display and recording work independently in task mode and with regenerated manifests/banks. Use a new video directory. Omitting --policy uses random actions for pipeline checks.

• The viewer uses the actual policy images: main view on the left, wrist views on the right, instructions and feedback in the sidebar.
• Each episode produces EPISODE.mp4 (composite), EPISODE.CAMERA_ID.mp4 for each full-resolution view, and JSON metadata. Videos share frame counts/order and control frequency; none contain UI/debug overlays.
• Up/Down scroll instructions, Space pauses, Tab shows a debug view excluded from policy input, and Esc aborts evaluation. Pauses neither advance physics nor duplicate recorded frames.
• --state-only-smoke disables images for task/instance checks and labels results accordingly; these are not RGB evaluation results. It cannot display or record. --render remains the separate native debugging viewer.

Camera contracts are now version 3.0. Old single-camera instances need regenerated runtime records and compatible instances, not merely a fresh draw. Source, TaskSpec and compiled-model identities remain checked. Existing PiPER-X mesh assets do not need downloading again: the common model assembly installs the new main sensor.

Tasks

task_id	Task
curling_push	Push a curling stone
button_indicator_matching	Match buttons and indicators
phone_lean	Lean a phone against a support
leveling_board1	Level a board with shims
rail_ball_to_cup	Roll a ball along a rail into a cup
pitch_pot	Toss into a pot
conveyor_grasp	Sort conveyor objects into bins
turntable_gamble1	Push a turntable through contact
ring_toss1	Toss rings
box_insertion	Insert a block into a slot
car_drive2	Drive using two physical joysticks
rope_hang	Rebalance a suspended chain
pan_flip	Flip food in a pan
laptop_into_drawer	Place a laptop in a drawer
find_object_in_drawer2	Find an object in drawers
tilte_maze	Control a tilting maze

Button episodes clamp the final score to [0, 100], including truncation. Running totals and signed ±20 rewards remain unchanged; raw_score retains the signed sum. Exam averages use individually clamped episode scores.

The phone table is 1.6 × 1.4 m, covering the arm base and mounting plate. Small supports span about 3.0–4.6 cm at their largest dimension.

Each phone_lean Task selects exactly two large and two small supports and varies only their initial positions between Episodes. Its three fixed candidate seeds each contain at least one bowl or bottle. No candidate or Episode seed is replaced to guarantee solvability; all four selected supports may score, and the phone angle must remain within 45–75 degrees for ten seconds.

Phone instances are maintained only in the main bank, react_manip/suites/piperx_candidates_v1/bank.json. For phone debugging, copy all three existing candidates and all 45 Episodes using the command below, then evaluate the resulting manifest. Selection copies existing records without resampling. --draw-seed remains in provenance, but this task-specific command does not use it to discard candidates. Retired banks and their old generator have been removed; historical versions remain available in Git history. Evaluation selection snapshots remain under outputs/ for reproducibility.

python scripts/select_task_instances.py --task phone_lean --draw-seed 42 \
  --out outputs/phone_selection
python scripts/evaluate_policy.py --manifest outputs/phone_selection/manifest.json \
  --policy my_policy:build --out outputs/phone_results.json

The phone has a black screen, a light-blue back, and a circular Home button on the lower half of the front. Keep the Home-button end down, back against the support, screen outward, and rounded bottom on the table. Upside-down, sideways, load-bearing screen contact, and angles above 75 degrees are rejected. After release with grasp assist disabled, angle, speed, and drift limits must hold continuously for ten seconds. Language rules come exclusively from the environment runtime interface; JSON contains no language copy.

For all 16 tasks, the versioned task_spec.json, instance_contract.json, and scoring implementation are authoritative for goals, scoring, failure/termination conditions, timing thresholds, and randomization ownership. The English instructions seen by policies come from env.py at reset and after every step.

The four closed-loop tasks above use the same parameter sampler in saved-bank evaluation and task-default 10/5 evaluation; bare single-scene debugging uses baseline Task parameters. Preserve every Episode's variation and runtime identity in result analysis so adaptation curves can be stratified by physical condition.

Optional manual Pygame control

# Pygame dependencies are included in requirements.txt.
python scripts/pygame_demo.py --task phone_lean

This tool only replays saved Task Instances and Episodes from the main bank; it does not resample or produce a protocol score. Use --instance/--episode to select exact records. N/P moves between Episodes and I/O between Task Instances. Arrow keys and PageUp/PageDown control translation; A/D, W/S, and Q/E control rotation. Space toggles the active gripper, Tab switches arms, and 1–9 or square brackets switch cameras. Backspace resets; Esc exits. Headless evaluation does not require Pygame.

For a temporary task-default scene selected by seed, a resolved Scenario, or embodiment debugging outside the fixed bank, use the separate development entry point:

python scripts/pygame_task_debug.py --task phone_lean --seed 0

pygame_task_debug.py accepts --robot and --scenario. It constructs an unscored debugging environment rather than replaying saved Task/Episode Instances; use pygame_demo.py when inspecting release-bank records or reproducibility.

Manual controls default to a target translation speed of 0.06 m/s and rotation speed of 0.3 rad/s (simulation time). Hold Shift for fast mode, or press F to toggle it: initially 1.0 m/s and 1.2 rad/s, useful for pushing or throwing. Use + / - to adjust the fast translation target by 0.05 m/s (0.05–2.0 m/s), or set it at launch with --fast-speed. The sidebar shows the active mode and target speed. For curling, approach in FINE mode, then use FAST for the push; adjust speed and release timing to the visible target. Actual speed depends on controller tracking and the task action limits. Releasing a movement key stops advancing that target axis and holds its release position; physical braking is not instantaneous. Losing window focus or switching arms clears movement input and holds the current poses. This tuning applies only to Pygame, not policy evaluation. On MacBook, PageUp/PageDown are Fn+Up/Fn+Down.

Run scripts/verify_curling_reachability.py to reproduce curling action-interface reachability checks. These use offline-calibrated diagnostic action replays, not learned-policy benchmark scores.

Layout and reproducibility

The release keeps only README.md and README.zh-CN.md as Markdown documentation. Versioned JSON contracts and implementation code are the source of truth for task rules, interfaces, and reproducibility constraints. Diagnostics are exposed directly through scripts/verify_*.py and scripts/audit_task_solvability.py, so released instructions do not depend on excluded auxiliary Markdown files.

• scripts/evaluate_all_tasks.py: complete 16-task evaluation entry point. evaluate_policy.py: lower-level bank/manifest/task evaluator. pygame_demo.py: fixed-bank manual replay. pygame_task_debug.py: temporary task-default scene debugging. build_agilex_assets.py and its helpers build robot assets.
• react_manip/: runner, protocols, observation filtering, cameras, robot integration, instance replay, and candidate selection.
• tasks/<task>/: environments, scoring/physics helpers, task declarations, and required assets.
• configs/: default profiles needed for task construction and parameter validation; not CLI overrides for the bank.
• react_manip/suites/piperx_candidates_v1/: the only maintained candidate bank, containing task instances and Episodes.
• assets/: scene textures and locally generated robot assets. Generated provenance.json records robot source versions and hashes.

Fixed instances validate source code, TaskSpecs, parameter contracts, compiled models, controllers, and dependency versions. Changing runtime Python/XML invalidates the recorded identity by design. Do not manually bypass validation by replacing hashes. Verified release migrations are recorded in generation.json. Write run outputs to new paths under outputs/; development tools and historical outputs are not distributed.

rope_hang starts with the chain draped over the bar. The task is to catch the slipping chain, redistribute its hanging lengths, release it, and maintain stable suspension. Its three candidates and 45 episodes were regenerated for this scene (TaskSpec 3.0 / contract 2.0). Old table-start selections must be redrawn; scores from the old and new scenes are not directly comparable.

Fixed dual PiPER-X conveyor_grasp uses two side tables. Bin centers are (1.25,0.50,0.76), (2.75,0.50,0.76), (1.25,0.90,0.76), and (2.75,0.90,0.76) meters, using the first two, three, or four bins according to category count. This layout avoids initial arm/base interference. Its three groups and 45 episodes were rebuilt (TaskSpec 3.0 / contract 2.0); do not reuse selections from the previous layout.

Python imports now use react_manip.* instead of benchmark.*, and the bank lives under react_manip/suites/piperx_candidates_v1/. There is no compatibility alias for the old package. Update external-policy imports and draw a new selection. The PiPER-X migration reconstructs initial simulation states with each task’s configured robot and camera contract 3.0. pip packaging remains future work.

Rebuild saved instances

The single active bank has 48 Task Instances and 720 Episodes, including three candidates and 45 Episodes for phone_lean; the four closed-loop parameter-adaptation tasks contribute 180 Episodes with explicit variations. Rebuilding compiles each Episode's resolved model, captures a fresh initial state, verifies its model/controller/source identity, restores the state exactly, and advances one zero-action step. This verifies replay compatibility, not robot task solvability. IDs and round plans are retained. Existing variations are preserved on subsequent rebuilds; old records without variations are migrated by the sampler. Generation reports record language samples and task-specific physical evidence.

python scripts/regenerate_instance_bank.py \
  --source react_manip/suites/piperx_candidates_v1/bank.json \
  --out outputs/rebuilt_piperx_bank

Use a new output directory. The script rebuilds task/episode records and the candidate manifest with linked digests. Evaluate its bank.json to draw a fresh 16-task manifest; do not reuse old selected manifests.

Language is owned by each task’s env.py through get_episode_language_prompt(). The adapter refreshes it after reset and every step and supplies info["language_prompt"] and info["language_prompt_structured"] to the policy. Structured fields separate task_instruction from round_instruction and round_index (zero-based, or null when no round is active). Missing runtime interfaces fail explicitly; task JSON is not a language fallback.

The drawer search task (find_object_in_drawer2) accepts either arm or both arms together. Both pose commands and both grippers are active. Its fixed rules, current-round instructions, constraints and end statuses are English-only; all five ordered placements still share 600 seconds and earn 20 points each.