EvoStudio Inference SDK

evostudio-inference-sdk is the model-integration boundary for EvoStudio's official simulation benchmarks. Users implement one Policy lifecycle and do not install simulator packages or configure benchmark scheduling and autotune parameters.

Install the SDK from PyPI:

python -m pip install evostudio-inference-sdk

import numpy as np

from evostudio_inference_sdk import ActionChunk, BenchmarkName, Policy, PolicyCapabilities, run


class UserPolicy(Policy):
    def load(self, context):
        self.model = load_model(context.checkpoint_path)
        return PolicyCapabilities(
            supported_benchmarks=(BenchmarkName.LIBERO,),
            required_features=("images.primary", "state.proprioception"),
            supported_suites={BenchmarkName.LIBERO: ("libero_10",)},
            supported_action_chunk_sizes=(1,),
        )

    def act(self, observation):
        return ActionChunk(np.asarray(self.model.predict(observation.features), dtype=np.float32))


run(UserPolicy())

EvoStudio launches this entrypoint once per selected inference replica, passes the staged checkpoint and benchmark contract to Policy.load(), and calls it through the local SDK protocol. The user process does not start a simulator or select evaluator concurrency.

The SDK also contains EvoStudio's platform benchmark runtime, durable work queue, process lifecycle, resource measurement, and automatic K/g selection under evostudio_inference_sdk.runtime. Those modules are platform internals and are not exported from the user-facing package API. Installing the PyPI package provides the public Policy API and protocol client; it does not install simulators, benchmark datasets, model frameworks, or the platform image dependencies needed to execute those internal runtime modules.

Earlier source releases exposed a libero extra, but it installed only imageio[ffmpeg]; it never installed a runnable LIBERO simulator or dataset. The extra has therefore been removed. Existing environments keep their already-installed packages, but consumers that use imageio directly must now declare imageio[ffmpeg]>=2.34,<3.0 in their own image or application dependencies. This is only a video dependency migration and is not a supported way to provision any Benchmark runtime.

The platform runtime uses three core scheduling concepts:

• Server: an inference process that loads one user Policy replica and serves action requests over a local socket. A Server does not run a simulator. Multiple Workers may share one Server, which can batch their concurrent requests. Autotune calls the selected Server replica count g.
• Worker: an evaluator process that owns a benchmark simulator lane. It claims Work Items, runs their environments and episodes, sends observations to a Server, applies the returned actions, and publishes results. A Worker does not load a separate copy of the user model. Autotune calls the selected Worker concurrency K.
• Work Item: the durable, atomically claimed unit of benchmark work. Its exact size is adapter-specific; for example, a RoboTwin Work Item is one task with its configured episodes. A failed Worker returns its claimed Work Item to the pending queue, while a completed Work Item is recorded idempotently so it cannot be scored twice.

RoboTwin and RoboCasa multi-episode work items publish each completed episode as an observable partial result. Product progress therefore advances while the item is still running, while official aggregation and durable completion continue to use only the item's final result. A non-OOM evaluator worker failure restarts the worker and replays its claimed item, with one retry allowed per runtime pass. Observable progress is recomputed from the durable queue snapshot and published through the atomic testlog.txt status file.

Runtime errors are collected in the append-only /root/testresult/errors.jsonl journal. It includes recoverable Worker failures as well as terminal checkpoint, Policy Server, simulator, configuration, and other evaluation errors. Worker records include the original traceback excerpt, and a terminal Worker failure carries that excerpt into the top-level error so Quick Test users see the simulator or Policy failure instead of only the Worker exit code. Component logs remain available for detailed diagnosis.

PolicyCapabilities declares the supported benchmarks, maximum inference batch size, and safe action-chunk sizes. Evaluator concurrency and replica count are platform-owned autotune parameters rather than user-declared capabilities. SDK-native LIBERO scored evaluations measure real Policy-backed episodes across K/g candidates and select the last candidate that improves throughput while resources remain safe. Candidates that lose Workers, report a Worker failure or OOM, or leave probe episodes incomplete are not selected.

API stability and versioning

The names exported by evostudio_inference_sdk.__all__ are the public SDK API. Modules below evostudio_inference_sdk.runtime are platform implementation details unless this document explicitly states otherwise; applications should not depend on their internal classes or command-line arguments.

The package follows Semantic Versioning. During the current 0.x Beta series, a minor release may make an incompatible change to __all__, the public Policy lifecycle, or the wire protocol, while patch releases remain compatible bug fixes. Every evaluation image pins one exact SDK version. User images should also pin the version they validated, and review release notes before upgrading to another minor version. The first 1.0.0 release will freeze the public Policy lifecycle and compatibility policy for stable use.

The installed version comes from distribution metadata and can be read with importlib.metadata.version("evostudio-inference-sdk"); evostudio_inference_sdk.__version__ is not part of the public API. The release owner updates the version in pyproject.toml and uv.lock, moves the CHANGELOG.md Unreleased entries into a dated release section, and records every user-visible API, Policy-lifecycle, and wire-protocol change in the same release merge request.

CHANGELOG.md is included in the source distribution, but is intentionally not copied into the wheel's runtime package. Wheel users can read the same release notes in the repository or PyPI source archive.

Episode-scoped state

The runtime calls reset(EpisodeContext) before an episode, sends its episode_id on each observation, and calls end_episode(episode_id) when the episode finishes. The SDK forwards this lifecycle but does not store or synchronize mutable Policy state.

Stateful Policies should isolate mutable state by episode_id and release it from end_episode(). Stateful and stateless Policy examples are provided in the EvoStudio user manual.

Model-owned input preprocessing

Each Benchmark sends the canonical observation defined by its Benchmark contract without model-specific resize, normalization, or channel/layout conversion. The user Policy performs those transformations before calling its model. Model input dimensions therefore do not belong in the public Benchmark or Policy API.

Current benchmark status

The repository contains the following platform execution chains. Their manifests describe dependencies that EvoStudio's Benchmark images or shared storage must provide; they are not dependencies of the PyPI distribution. A chain being present here does not guarantee that every deployment has provisioned its external artifacts.

LIBERO and CALVIN convert their native observations to the canonical SDK feature names and send native 7-dimensional actions back to their simulators. CALVIN's task-D-to-task-D path implements the official 1,000 long-horizon sequences with static/gripper images, proprioception and language. The suite list and runtime paths for every chain remain owned by its installed Benchmark image manifest.

• Evaluation has two product paths. Custom-image users provide their own SDK Policy and saved image. StarVLA users may instead upload a standard run snapshot and directly select the official StarVLA image. Both paths use platform Benchmark environments and evaluators from /autodl-fs.

The complete LIBERO entrypoint and scoring contract is documented in LIBERO_EVALUATION_PROTOCOL.md in the Evaluation Kit repository. The LIBERO document defines the /root/startbenchmark.sh arguments, checkpoint staging, canonical observation/action mapping, episode initialization and horizons, success detection, output files, error codes, and required validation sequence. The SDK-native CALVIN manifest, observation/action mapping, official long-horizon evaluator, aggregation, and validation procedure are documented in CALVIN_EVALUATION_PROTOCOL.md in the Evaluation Kit repository.

The SDK integration, autotune, and benchmark-runtime throughput results are maintained in the Evaluation Kit repository. The report keeps LIBERO and RoboTwin results separate and includes the StarVLA official four-GPU RoboTwin launcher baseline used for the runtime comparison.

Local policy HIL worker

evostudio-inference-worker is the independent model process used by policy_hil_continuous collection. It reuses the SDK Unix-socket wire framing, binds no network address, and never opens robot or camera devices. Local checkpoints and all referenced base/tokenizer models must be inside explicitly allowlisted roots:

python -m evostudio_inference_sdk.runtime.hil.worker \
  --socket /run/evostudio-inference/worker.sock \
  --checkpoint-root outputs=/home/dell/outputs \
  --dependency-root /home/dell/models \
  --hash-cache /var/lib/evostudio-inference/checkpoint-hashes.json \
  --device cuda:0 \
  --torch-threads 4 \
  --gpu-memory-fraction 0.8 \
  --worker-git-commit <evaluation-kit-commit> \
  --policy-runtime-version <pinned-lerobot-runtime>

The first runtime declares support only for complete Pi05 checkpoints. Catalog discovery does not trust browser-provided paths. Loading verifies checkpoint and dependency hashes, validates observation/action schemas, runs a real dry-run, and freezes the selected session. The worker accepts one inference request at a time and rejects stale policy epochs so an action chunk generated before an intervention cannot be resumed afterward.

The policy environment is part of checkpoint compatibility. The Pi05 HIL worker requires the pinned LeRobot 0.6.0 runtime used by the validated native RTC deployment; archived evostudio-engine checkouts are not a supported runtime source. Install that runtime into the worker environment and record its immutable build identity with --policy-runtime-version. Startup verifies the installed distribution version and native RTC module instead of trusting the label or trying a second loader. A successful real dry-run freezes that runtime version into the session provenance.

The HIL v4 contract runs Pi05 with real-time chunking only. It preserves the pinned native LeRobot RTC timing semantics across the process boundary. The Client reports the remaining action queue depth with each observation. The Worker tracks model-local inference latency, uses its historical maximum for RTC prefix guidance, and returns both the complete action chunk and the current model-local merge delay. At the control-frame boundary, the Client drops that reported number of actions before replacing its queue. End-to-end action age is still measured independently for fail-closed safety. Takeover and handback reset both sides of the queue, so actions generated before an intervention cannot resume.

The example systemd unit and environment file are in deploy/systemd. The unit gives the worker lower CPU/IO priority than the client, exposes only a group-readable Unix socket, permits only Unix-domain networking, and restarts the process after failure. A restarted worker returns to idle; robot motion still requires a new client preflight and explicit episode start. Runtime logs stay in journald (journalctl -u evostudio-inference-worker), readiness is available only through the local socket, and hash/cache state stays under /var/lib/evostudio-inference.

Direct evaluation with the official StarVLA image

Custom images contain the user's SDK Policy, dependencies and /workspace/evostudio-policy.json. The separate official StarVLA image path uses an EvoStudio-published image with a pinned StarVLA runtime and a preconfigured evaluation entrypoint:

/opt/evostudio/starvla/starvla_official_policy.py

Cloud stages the user-selected StarVLA run directory. The run keeps StarVLA's native layout:

<run>/
├── config.yaml
├── dataset_statistics.json
├── checkpoints/
│   └── <selected full checkpoint>
└── <local resources referenced by config.yaml>

The staged evaluation snapshot contains one selected full checkpoint. SessionContext.checkpoint_path may point to that checkpoint file or to the run directory. The official loader deterministically locates the checkpoint and adjacent config/statistics, builds the framework from config.yaml, performs strict state-dict loading, and initializes training-time normalization.

The official Policy follows the pinned StarVLA reference client for the active Benchmark. RoboTwin, RoboCasa, and LIBERO+ use OpenCV INTER_AREA; LIBERO uses PIL bilinear resize; CALVIN uses the upstream aspect-preserving resize-and-pad and uint8 conversion. All produce the pinned clients' 224x224 StarVLA model input without changing the Benchmark's canonical Observation. The Policy does not derive this client-side preprocessing from, or rewrite, the uploaded StarVLA configuration. Tokenization, inference kwargs and dtype are read from the StarVLA configuration or its framework defaults. Benchmark environments, datasets, evaluator workers and native action application are provided by /autodl-fs. Both custom images and the official StarVLA image support an independent real K=1/g=1 quick test and full formal evaluation.

config.yaml must carry an inference section. It names the Observation features the Policy consumes and declares the per-dimension action transforms:

inference:
  dtype: bfloat16                    # required: bfloat16, float16 or float32
  unnorm_key: franka                 # optional: training normalization statistics key
  inputs:                            # required
    images:                          # required, non-empty; order is the camera order
      - images.primary
      - images.wrist
    language: language.instruction   # required
    state: state.proprioception      # optional, declare only if the model consumes it
  kwargs:                            # optional, forwarded to the StarVLA framework
    do_sample: false
    use_ddim: true
    num_ddim_steps: 10
  action_transforms:                 # optional, applied in order to the model output
    - index: 6
      operation: threshold
      threshold: 0.5
      above: -1.0
      below: 1.0

inputs values are Benchmark Observation feature names; LIBERO exposes images.primary, images.wrist, state.proprioception and language.instruction. A missing inputs or malformed action_transforms entry fails Policy startup. A syntactically valid but semantically wrong camera order or action transform cannot be detected automatically, so both must match the training-time definitions.

action_transforms supports threshold (index, threshold, above, below) and affine (index, scale, offset). The example above reproduces the LIBERO gripper binarization: dimension 6 becomes -1.0 when the raw value exceeds 0.5, and 1.0 otherwise.

The image ships an exporter that produces a portable run snapshot:

/root/miniconda3/envs/starVLA/bin/python \
  /opt/evostudio/starvla/export_starvla_eval_run.py \
  --run-dir <training run> \
  --checkpoint <training run>/checkpoints/<selected checkpoint> \
  --output <export directory> \
  --resource-root <root of the local construction resources>

The exporter copies every local resource referenced by the framework and inference sections into the export directory, rewrites those configuration values to relative paths, and runs a strict-load preflight. A hand-assembled snapshot that keeps training-host absolute paths (for example framework.qwenvl.base_vlm) cannot be resolved inside the evaluation instance.

The official image Policy process contract is:

{
  "schema_version": 1,
  "command": [
    "/root/miniconda3/envs/starVLA/bin/python",
    "/opt/evostudio/starvla/starvla_official_policy.py"
  ],
  "cwd": "/root/starVLA"
}

The LIBERO platform entrypoint loads the Policy process contract from the fixed image path /workspace/evostudio-policy.json:

{
  "schema_version": 1,
  "command": ["/opt/user/bin/python", "/workspace/user_policy.py"],
  "cwd": "/workspace"
}

EvoStudio selects the Policy runtime and validates the file independently during quick test and scored evaluation. End users do not configure the path, K/g, or benchmark runtime flags.

License

Copyright 2026 EvoMind Team. Licensed under the Apache License, Version 2.0. See LICENSE.