LIBERO-PRO-Benchmark

A PyTorch-free simulation benchmark for evaluating VLA policies on LIBERO and LIBERO-PRO, with the original benchmark imports and bundled evaluation resources.

What this package provides

libero-pro-benchmark is an independent, modernized extraction of the benchmark from LIBERO-PRO, which builds on LIBERO. It provides task definitions, MuJoCo environments, success checks, initial states, and the upstream perturbation utilities. It is not an official upstream release.

One installation supports both LIBERO and LIBERO-PRO evaluation. You do not need to install the original repositories alongside it. Existing evaluation code can keep imports such as from libero.libero import benchmark and from libero.libero.envs import OffScreenRenderEnv.

The benchmark does not install PyTorch, a VLA model, or a training framework. Supply your own policy and its inference dependencies. There is no CPU, CUDA, macOS, or Linux extra to select: the policy's device is controlled by the calling application.

Model assets, task descriptions, and evaluation initial states are installed with the package. This includes the official LIBERO-PRO evaluation resources that upstream distributes separately; no tasks have been invented or added to the upstream registry. No manual resource copying or first-use download is needed for the covered tasks.

Why this package?

A VLA evaluator needs the simulation environment, task instructions, initial states, and success checks. The upstream LIBERO-PRO repository also includes policy training code and setup instructions tied to older Python and PyTorch releases. Recreating that full stack can add unnecessary dependencies and make integration with a policy's own environment harder.

This package makes the benchmark usable as a normal dependency in an existing evaluation project:

• Simpler installation. Install with pip install libero-pro-benchmark or uv add libero-pro-benchmark. Model assets and the official separately distributed evaluation resources are installed automatically for the covered tasks.
• Independent policy dependencies. The benchmark uses NumPy initial states and does not require PyTorch. Your policy can choose its own inference framework and device without inheriting the upstream training stack.
• Modern simulation setup. Use supported current Python and MuJoCo versions within the declared constraints, or select compatible for the measured reference dependency stack. Simulator upgrades can change trajectories and rendering.
• Less evaluation-code migration. Retain the original libero.libero imports and benchmark APIs. Task definitions and ordering are preserved, and initial-state conversion preserves the original numeric bytes.
• LIBERO and LIBERO-PRO together. One benchmark dependency provides the five standard LIBERO suites and the covered PRO extensions, with explicit resource-availability checks for other upstream entries.

It is intended for researchers who already have a policy or evaluation runner and need reusable benchmark environments and tasks. The contribution is installation, dependency separation, and resource packaging; it does not introduce a new benchmark protocol, new tasks, or a claim of identical physics across simulator versions.

Install

Use a virtual environment with CPython 3.10–3.14. The project targets macOS and Linux. Native installation and rendering have been verified on macOS / Apple Silicon; Linux still awaits real-machine validation.

With pip, in your active environment:

python -m pip install libero-pro-benchmark

Or add it to your own uv project:

uv add libero-pro-benchmark

The main package automatically installs five required libero-pro-benchmark-assets-* packages. You do not install these individually or choose a subset. The model resource wheels total approximately 340 MiB, in addition to the benchmark and simulation dependencies. After installation, evaluation resources are available offline.

Use a clean environment when replacing upstream LIBERO or LIBERO-PRO. Remove the upstream distribution from your project's dependencies before adding this package; they share the libero import namespace and must not be used together. Your policy may still use PyTorch in the same environment.

Reference simulation dependencies (compatible)

The default installation allows newer MuJoCo and NumPy versions within the package constraints. For the measured reference stack, install the compatible extra in a Python 3.10 or 3.11 environment; Python 3.11 is recommended:

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install "libero-pro-benchmark[compatible]"

For a uv project targeting Python 3.10/3.11:

uv add "libero-pro-benchmark[compatible]"

Or install directly into a uv-managed Python 3.11 virtual environment:

uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python "libero-pro-benchmark[compatible]"

Quote package names containing brackets when using shells such as zsh. The extra pins MuJoCo 2.3.7, NumPy 1.26.4, and OpenCV below 4.12. It does not change your Python interpreter. MuJoCo 2.3.7 has no Python 3.12+ wheels; use a separate Python 3.11 environment rather than trying to install this stack into Python 3.14. For uv projects, keep requires-python consistent with the chosen interpreter range, for example ">=3.11,<3.12" for a Python 3.11 evaluation project.

Initial-state arrays preserve the original dtype, shape, ordering, and numeric bytes in either installation. The default simulator can produce different trajectories and images from the old stack. Reference-stack equality was measured on specific same-host trajectory and rendering samples; compatible is not a guarantee of identical results on every machine or every rollout. Record your Python and simulation dependency versions when reporting evaluation results.

Which tasks are included?

Standard LIBERO

All tasks in the five standard suites are included, with their upstream names and base ordering:

Suite name	Task entries	Evaluation focus
libero_spatial	10	Spatial relations
libero_object	10	Object variation
libero_goal	10	Goal variation
libero_90	90	The LIBERO-90 task collection
libero_10	10	The LIBERO-10 task collection

These are 130 suite entries; suite counts are not a claim of 130 mutually distinct underlying tasks.

LIBERO-PRO

The four base suites libero_spatial, libero_object, libero_goal, and libero_10 each have the following resource-complete PRO variants:

Suffix	Perturbation	Example suite	Entries per variant suite
_lan	Language instruction	libero_spatial_lan	10
_object	Object replacement	libero_spatial_object	10
_swap	Object-position swaps	libero_spatial_swap	10
_task	Task replacement	libero_spatial_task	10

This gives 16 PRO suites and 160 entries in addition to the standard suites. To evaluate one, select its exact suite name using the same API shown below. The perturbation utilities also retain upstream environment, object, language, swap, and task transformations; generating a custom task does not make it an official evaluation task or supply official initial states for it.

Other registered upstream extensions

The complete upstream registry is retained: 83 suites and 960 task entries. Registration does not mean all required resources exist. The bundled inventory identifies 33 suites with complete known resources and 409 entries with the known required task, initialization, and model resources. The first task of each of the 33 complete suites passed reset, state restoration, and stepping; this does not certify every rollout or a policy's success rate.

Besides the standard and 16 PRO suites above, the complete suites are libero_mine, libero_study_table, libero_object_two_normal_episodes; the _with_mug variants of libero_spatial, libero_object, libero_goal, and libero_10; the _with_yellow_book variants of libero_spatial, libero_object, and libero_goal; and the _with_milk variants of libero_spatial and libero_goal.

Other upstream extensions have missing or empty initial states, missing task descriptions, or unavailable models. The missing resources were supplied by neither the recorded upstream checkout nor its official evaluation dataset, and are not fabricated here. Do not treat every registered extension, including the _env family, as runnable.

Check the installed resource coverage before choosing an extension:

libero-pro list
libero-pro audit --output resource-availability.json

In a uv project, prefix these commands with uv run. The list reports entries with required resources for each suite; the audit identifies missing resources for individual tasks. These commands do not run your policy. audit --strict intentionally fails when any registered entry lacks resources.

Run an environment

This executable example restores the first official initial state and performs a short zero-action rollout. The zero actions demonstrate the API; they are not a task-solving policy or an evaluation protocol.

from pathlib import Path

import numpy as np
from libero.libero import benchmark, get_libero_path
from libero.libero.envs import OffScreenRenderEnv

suite_name = "libero_spatial"  # e.g. "libero_spatial_lan" for a PRO suite
suite = benchmark.get_benchmark(suite_name)(task_order_index=0)
task_id = 0
task = suite.get_task(task_id)
initial_states = suite.get_task_init_states(task_id)
bddl_file = (
    Path(get_libero_path("bddl_files")) / task.problem_folder / task.bddl_file
)

env = OffScreenRenderEnv(
    bddl_file_name=str(bddl_file),
    camera_heights=128,
    camera_widths=128,
)
try:
    env.seed(123)
    env.reset()
    obs = env.set_init_state(initial_states[0])
    print("Instruction:", task.language)
    for step in range(20):
        # Replace with your policy's action using obs and task.language.
        action = np.zeros(env.env.action_dim)
        obs, reward, done, info = env.step(action)
        success = bool(env.check_success())
        if success or done:
            break
    print("Success:", success)
finally:
    env.close()

The standard Panda controller uses a seven-element action. Observations include agentview_image, robot0_eye_in_hand_image, and robot state fields. task.language is the task instruction to pass to a language-conditioned policy. Keep the observation preprocessing, action representation, task order, initial-state selection, and rollout horizon expected by your chosen VLA evaluation protocol. Task success is available through env.check_success(); do not assume every episode ending means success.

The original four-value step() return is preserved: (obs, reward, done, info). It is not Gymnasium's five-value return format. Vector environment and segmentation imports are also retained under libero.libero.envs.

The alternate imports from libero_pro_benchmark import benchmark, get_libero_path and from libero_pro_benchmark.envs import OffScreenRenderEnv expose the same implementation objects. The installation name is libero-pro-benchmark; the original namespace remains libero.libero.

For a quick installation check, without a policy:

libero-pro smoke --suite libero_spatial --steps 2
libero-pro smoke --suite libero_spatial_lan --steps 2

Rendering needs a working graphics backend. On headless Linux with EGL drivers installed, set MUJOCO_GL=egl before starting Python. macOS uses its native backend; do not copy the Linux EGL setting into a macOS environment. libero-pro smoke --no-render checks state simulation without camera rendering.

Use it with an existing VLA evaluator

Replace the upstream benchmark dependency with libero-pro-benchmark, retain the original imports, and provide your evaluator's own model/inference dependencies. Code using the public task and environment APIs can continue to use them without changing those imports.

There are explicit migration differences for code bypassing the APIs:

• Initial states are stored as losslessly converted NPZ arrays. Use suite.get_task_init_states(task_id); it returns a NumPy array. Code calling torch.load() on an initialization file directly must change its loading call.
• Use get_libero_path("assets"), get_libero_path("bddl_files"), and get_libero_path("init_states") for installed resources. Paths constructed relative to an upstream checkout or a module's __file__ are not portable to this installation.
• Upstream policy training code and its training loaders are not included. This package supplies the benchmark environment for your evaluator.
• The old implicit ~/.libero configuration is not read. For explicit resource overrides, set LIBERO_CONFIG_PATH to a directory containing a config.yaml. An initial-state override must contain NPZ-format states.

The package does not implement a particular model service, evaluation leaderboard, or universal VLA adapter. Your evaluator remains responsible for inference, episode scheduling, metrics, and its benchmark protocol.

Optional demonstrations and video tools

Neither extra is needed for ordinary online evaluation:

Extra	What it enables	Installation
datasets	h5py and the upstream get_dataset_info HDF5 inspection helper	python -m pip install "libero-pro-benchmark[datasets]"
video	The upstream VideoWriter with imageio and FFmpeg	python -m pip install "libero-pro-benchmark[video]"

Extras can be combined, for example "libero-pro-benchmark[compatible,video]" in Python 3.11. In uv projects, use the same requirement with uv add.

The official LIBERO demonstration dataset contains HDF5 demonstration trajectories for training or analysis. Installing datasets does not download it and does not add a training framework. The official LIBERO-PRO evaluation dataset supplies task descriptions and initial states, rather than HDF5 demonstrations; the evaluation resources covered here are already bundled.

Installing video does not record automatically. Callers explicitly create libero.libero.utils.video_utils.VideoWriter with save_video=True, append observations or frames, and save the video. Camera observations are available without this extra, so you may use your own encoder instead.

Attribution

This independent package preserves the upstream license statements: original code is MIT-licensed, the upstream dataset is labeled CC BY 4.0, and upstream asset ownership and licensing continue to apply. Cite the LIBERO paper and LIBERO-PRO paper when using the benchmark in research.