EvoStudio Inference SDK

evostudio-inference-sdk is the model-integration boundary for EvoStudio's official simulation benchmarks. Users implement one Policy lifecycle and do not install simulator packages or configure benchmark scheduling and autotune parameters.

Install the SDK from PyPI:

python -m pip install evostudio-inference-sdk

import numpy as np

from evostudio_inference_sdk import ActionChunk, BenchmarkName, Policy, PolicyCapabilities, run


class UserPolicy(Policy):
    def load(self, context):
        self.model = load_model(context.checkpoint_path)
        return PolicyCapabilities(
            supported_benchmarks=(BenchmarkName.LIBERO,),
            required_features=("images.primary", "state.proprioception"),
            supported_suites={BenchmarkName.LIBERO: ("libero_10",)},
            supported_action_chunk_sizes=(1,),
        )

    def act(self, observation):
        return ActionChunk(np.asarray(self.model.predict(observation.features), dtype=np.float32))


run(UserPolicy())

EvoStudio launches this entrypoint once per selected inference replica, passes the staged checkpoint and benchmark contract to Policy.load(), and calls it through the local SDK protocol. The user process does not start a simulator or select evaluator concurrency.

The SDK also contains EvoStudio's platform benchmark runtime, durable work queue, process lifecycle, resource measurement, and automatic K/g selection under evostudio_inference_sdk.runtime. Those modules are platform internals and are not exported from the user-facing package API. Installing the PyPI package provides the public Policy API and protocol client; it does not install simulators, benchmark datasets, model frameworks, or the platform image dependencies needed to execute those internal runtime modules.

Earlier source releases exposed a libero extra, but it installed only imageio[ffmpeg]; it never installed a runnable LIBERO simulator or dataset. The extra has therefore been removed. Existing environments keep their already-installed packages, but consumers that use imageio directly must now declare imageio[ffmpeg]>=2.34,<3.0 in their own image or application dependencies. This is only a video dependency migration and is not a supported way to provision any Benchmark runtime.

The platform runtime uses three core scheduling concepts:

• Server: an inference process that loads one user Policy replica and serves action requests over a local socket. A Server does not run a simulator. Multiple Workers may share one Server, which can batch their concurrent requests. Autotune calls the selected Server replica count g.
• Worker: an evaluator process that owns a benchmark simulator lane. It claims Work Items, runs their environments and episodes, sends observations to a Server, applies the returned actions, and publishes results. A Worker does not load a separate copy of the user model. Autotune calls the selected Worker concurrency K.
• Work Item: the durable, atomically claimed unit of benchmark work. Its exact size is adapter-specific; for example, a RoboTwin Work Item is one manifest-sized episode window from a task. Windows preserve the task's canonical seed-cache indices while allowing idle Workers to share long-horizon tasks. A failed Worker returns its claimed Work Item to the pending queue, while a completed Work Item is recorded idempotently so it cannot be scored twice.

RoboTwin and RoboCasa multi-episode work items publish each completed episode as an observable partial result. Product progress therefore advances while the item is still running, while official aggregation and durable completion continue to use only the item's final result. A non-OOM RoboTwin native evaluator failure with a positive exit code is recorded as a terminal failed item (episodes completed before the crash are kept and the remainder of that window counted as failures), so the worker continues with the next item instead of aborting the evaluation pass; OOM and signal-terminated exits still restart the worker and replay its claimed item, with one retry allowed per runtime pass. Observable progress is recomputed from the durable queue snapshot and published through the atomic testlog.txt status file.

Runtime errors are collected in the append-only /root/testresult/errors.jsonl journal. It includes recoverable Worker failures as well as terminal checkpoint, Policy Server, simulator, configuration, and other evaluation errors. A failed RoboTwin window keeps the native log excerpt in its worker and native logs and counts that window's missing episodes as failures in the official aggregate, so Quick Test users see the simulator or Policy failure instead of only a Worker exit code. Component logs remain available for detailed diagnosis.

PolicyCapabilities declares the supported benchmarks, maximum inference batch size, safe action-chunk sizes, and optional per-Benchmark action representations:

PolicyCapabilities(
    supported_benchmarks=(BenchmarkName.ROBOTWIN,),
    required_features=("images.head", "state.joints"),
    supported_action_chunk_sizes=(37,),
    action_representations={BenchmarkName.ROBOTWIN: "robotwin.ee-compact-quat-v1"},
)

The runtime validates each declared representation against the Benchmark specification and requires every loaded replica to resolve the same action contract before evaluation starts. Existing Policies that omit action_representations resolve to the Benchmark default. The resolved value and its source are recorded in inference metadata.

Evaluator concurrency and replica count are platform-owned autotune parameters rather than user-declared capabilities. SDK-native scored evaluations measure real Policy-backed episodes across K/g candidates. Every candidate runs the same deterministic, representative episode set. The minimum safe K establishes a success-rate baseline, and later candidates must retain at least 85% of that baseline while improving throughput and keeping resources safe. Candidates that regress beyond this 15% tolerance, lose Workers, report a Worker failure or OOM, or leave probe episodes incomplete are rejected. The platform stops a faulty probe immediately, limits each candidate to 15 minutes and the complete tune to 30 minutes, then continues with the last healthy K/g pair. /root/testresult/autotune/selection.json is updated atomically when every candidate starts, advances, or stops, so running, accepted, rejected, and timed-out trials remain visible to platform log synchronization without contributing episodes to the formal score.

API stability and versioning

The names exported by evostudio_inference_sdk.__all__ are the public SDK API. Modules below evostudio_inference_sdk.runtime are platform implementation details unless this document explicitly states otherwise; applications should not depend on their internal classes or command-line arguments.

The package follows Semantic Versioning. During the current 0.x Beta series, a minor release may make an incompatible change to __all__, the public Policy lifecycle, or the wire protocol, while patch releases remain compatible bug fixes. Every evaluation image pins one exact SDK version. User images should also pin the version they validated, and review release notes before upgrading to another minor version. The first 1.0.0 release will freeze the public Policy lifecycle and compatibility policy for stable use.

The installed version comes from distribution metadata and can be read with importlib.metadata.version("evostudio-inference-sdk"); evostudio_inference_sdk.__version__ is not part of the public API. The release owner updates the version in pyproject.toml and uv.lock, moves the CHANGELOG.md Unreleased entries into a dated release section, and records every user-visible API, Policy-lifecycle, and wire-protocol change in the same release merge request.

CHANGELOG.md is included in the source distribution, but is intentionally not copied into the wheel's runtime package. Wheel users can read the same release notes in the repository or PyPI source archive.

Episode-scoped state

The runtime calls reset(EpisodeContext) before an episode, sends its episode_id on each observation, and calls end_episode(episode_id) when the episode finishes. The SDK forwards this lifecycle but does not store or synchronize mutable Policy state.

Stateful Policies should isolate mutable state by episode_id and release it from end_episode(). Stateful and stateless Policy examples are provided in the EvoStudio user manual.

Model-owned input preprocessing

Each Benchmark sends the canonical observation defined by its Benchmark contract without model-specific resize, normalization, or channel/layout conversion. The user Policy performs those transformations before calling its model. Model input dimensions therefore do not belong in the public Benchmark or Policy API.

Current benchmark status

The repository contains the following platform execution chains. Their manifests describe dependencies that EvoStudio's Benchmark images or shared storage must provide; they are not dependencies of the PyPI distribution. A chain being present here does not guarantee that every deployment has provisioned its external artifacts.

LIBERO and CALVIN convert their native observations to the canonical SDK feature names and send native 7-dimensional actions back to their simulators. CALVIN's task-D-to-task-D path implements the official 1,000 long-horizon sequences with static/gripper images, proprioception and language. The suite list and runtime paths for every chain remain owned by its installed Benchmark image manifest.

• Evaluation has two product paths. Custom-image users provide their own SDK Policy and saved image. StarVLA users may instead upload a standard run snapshot and directly select the official StarVLA image. Both paths use platform Benchmark environments and evaluators from /autodl-fs.

The complete LIBERO entrypoint and scoring contract is documented in LIBERO_EVALUATION_PROTOCOL.md in the Evaluation Kit repository. The LIBERO document defines the /root/startbenchmark.sh arguments, checkpoint staging, canonical observation/action mapping, episode initialization and horizons, success detection, output files, error codes, and required validation sequence. The SDK-native CALVIN manifest, observation/action mapping, official long-horizon evaluator, aggregation, and validation procedure are documented in CALVIN_EVALUATION_PROTOCOL.md in the Evaluation Kit repository.

The SDK integration, autotune, and benchmark-runtime throughput results are maintained in the Evaluation Kit repository. The report keeps LIBERO and RoboTwin results separate and includes the StarVLA official four-GPU RoboTwin launcher baseline used for the runtime comparison.

Policy HIL model components

The SDK supplies checkpoint discovery, immutable verification, Pi0.5 loading, processor construction, and the pinned LeRobot RTC configuration used by policy_hil_continuous. It does not provide a robot-control service and never owns CAN, cameras, recording, progress, or uploads.

The separately packaged evostudio-hil-runtime service in the EvoStudio Client repository imports load_pi05_runtime. That service keeps the official RTCInferenceEngine, model, processor pipelines, four arms, three cameras, and pedal in one Python process. There is no action-chunk wire protocol or Client queue adapter.

Local checkpoints and referenced model resources must remain inside explicitly allowlisted roots. The caller obtains a VerifiedCheckpoint from CheckpointCatalog, freezes its hashes into the collection session, and passes it to the component factory. The factory also requires an explicit local tokenizer path because deployed checkpoints can contain training-host paths that are not valid on the collection machine.

The runtime requires the pinned LeRobot 0.6.0 installation used by the validated native Pi0.5 RTC deployment. Archived evostudio-engine checkouts are not a runtime source. The policy_hil_runtime/v1 contract belongs to the Client HIL service; this package exposes model components rather than a socket server.

Direct evaluation with the official StarVLA image

Custom images contain the user's SDK Policy, dependencies and /workspace/evostudio-policy.json. The separate official StarVLA image path uses an EvoStudio-published image with a pinned StarVLA runtime and a preconfigured evaluation entrypoint:

/opt/evostudio/starvla/starvla_official_policy.py

Cloud stages the user-selected StarVLA run directory. The run keeps StarVLA's native layout:

<run>/
├── config.yaml
├── dataset_statistics.json
├── checkpoints/
│   └── <selected full checkpoint>
└── <local resources referenced by config.yaml>

The staged evaluation snapshot contains one selected full checkpoint. SessionContext.checkpoint_path may point to that checkpoint file or to the run directory. The official loader deterministically locates the checkpoint and adjacent config/statistics, builds the framework from config.yaml, performs strict state-dict loading, and initializes training-time normalization.

The official Policy follows the pinned StarVLA reference client for the active Benchmark. RoboTwin, RoboCasa, and LIBERO+ use OpenCV INTER_AREA; LIBERO uses PIL bilinear resize; CALVIN uses the upstream aspect-preserving resize-and-pad and uint8 conversion. All produce the pinned clients' 224x224 StarVLA model input without changing the Benchmark's canonical Observation. The Policy does not derive this client-side preprocessing from, or rewrite, the uploaded StarVLA configuration. Tokenization, inference kwargs and dtype are read from the StarVLA configuration or its framework defaults. Benchmark environments, datasets, evaluator workers and native action application are provided by /autodl-fs. Both custom images and the official StarVLA image support an independent real K=1/g=1 quick test and full formal evaluation.

config.yaml must carry an inference section. It names the Observation features the Policy consumes and declares the per-dimension action transforms:

inference:
  dtype: bfloat16                    # required: bfloat16, float16 or float32
  unnorm_key: franka                 # optional: training normalization statistics key
  inputs:                            # required
    images:                          # required, non-empty; order is the camera order
      - images.primary
      - images.wrist
    language: language.instruction   # required
    state: state.proprioception      # optional, declare only if the model consumes it
  kwargs:                            # optional, forwarded to the StarVLA framework
    do_sample: false
    use_ddim: true
    num_ddim_steps: 10
  action_transforms:                 # optional, applied in order to the model output
    - index: 6
      operation: threshold
      threshold: 0.5
      above: -1.0
      below: 1.0

inputs values are Benchmark Observation feature names; LIBERO exposes images.primary, images.wrist, state.proprioception and language.instruction. A missing inputs or malformed action_transforms entry fails Policy startup. A syntactically valid but semantically wrong camera order or action transform cannot be detected automatically, so both must match the training-time definitions.

action_transforms supports threshold (index, threshold, above, below) and affine (index, scale, offset). The example above reproduces the LIBERO gripper binarization: dimension 6 becomes -1.0 when the raw value exceeds 0.5, and 1.0 otherwise.

The image ships an exporter that produces a portable run snapshot:

/root/miniconda3/envs/starVLA/bin/python \
  /opt/evostudio/starvla/export_starvla_eval_run.py \
  --run-dir <training run> \
  --checkpoint <training run>/checkpoints/<selected checkpoint> \
  --output <export directory> \
  --resource-root <root of the local construction resources>

The exporter copies every local resource referenced by the framework and inference sections into the export directory, rewrites those configuration values to relative paths, and runs a strict-load preflight. A hand-assembled snapshot that keeps training-host absolute paths (for example framework.qwenvl.base_vlm) cannot be resolved inside the evaluation instance.

The official image Policy process contract is:

{
  "schema_version": 1,
  "command": [
    "/root/miniconda3/envs/starVLA/bin/python",
    "/opt/evostudio/starvla/starvla_official_policy.py"
  ],
  "cwd": "/root/starVLA"
}

The LIBERO platform entrypoint loads the Policy process contract from the fixed image path /workspace/evostudio-policy.json:

{
  "schema_version": 1,
  "command": ["/opt/user/bin/python", "/workspace/user_policy.py"],
  "cwd": "/workspace"
}

EvoStudio selects the Policy runtime and validates the file independently during quick test and scored evaluation. End users do not configure the path, K/g, or benchmark runtime flags.

License

Copyright 2026 EvoMind Team. Licensed under the Apache License, Version 2.0. See LICENSE.