LIBERO-PRO-Benchmark

A modern, PyTorch-free benchmark for LIBERO and LIBERO-PRO VLA evaluation, with uv setup, original benchmark APIs, and lossless initial-state conversion.

A standalone LIBERO-PRO simulation and task package managed with uv. It provides environments, tasks, success predicates, initial states, and perturbation tools for external policy evaluation. Policy inference and training belong in the caller's project. Benchmark execution does not require PyTorch or CPU/CUDA extras.

One installation supports both LIBERO and LIBERO-PRO evaluation. This package includes all tasks in the five standard LIBERO suites (libero_spatial, libero_object, libero_goal, libero_90, and libero_10), with their task names and base ordering preserved, alongside the LIBERO-PRO benchmark extensions. You do not need to install the original LIBERO and LIBERO-PRO projects separately.

The package is based on upstream commit eafdb809426b13153aa1e4c42d6601844217dfec. It preserves all 83 registered suites, 960 task entries, and their original ordering.

This package bundles the official evaluation resources that LIBERO-PRO publishes separately, so users do not need to download and copy them manually. The bundled additions comprise 160 BDDL task files and 160 initialization files for tasks already registered upstream; no new tasks were added, and existing files were not overwritten. Initialization arrays are packaged as losslessly converted NPZ files.

The five standard LIBERO suites and all 16 PRO suites with _lan, _object, _swap, or _task suffixes have complete resources in the audited inventory. Remaining gaps concern other registered upstream extensions whose required resources were not supplied by either the source checkout or the official dataset at the recorded revisions. These extensions cannot run as supplied; this package does not invent replacement resources. See the resource inventory and validation scope.

Installation

cd LIBERO-PRO-Benchmark
uv sync
uv run libero-pro list
uv run libero-pro smoke --no-render
uv run libero-pro smoke

Dependency ranges are defined in pyproject.toml; exact resolved versions are recorded in uv.lock. uv sync installs the committed resolution and does not automatically adopt newly released dependencies. To update to newer versions within the declared constraints:

uv lock --upgrade
uv sync

Run the manual validation checks after an upgrade. The version numbers in validation reports describe measured dependency snapshots, not a promise about future releases. .python-version sets the default interpreter; requires-python declares the supported range. Use a recent uv release that recognizes the interpreter specified by this checkout.

No --extra mac, --extra linux, or --extra cuda is needed.

Compatibility baseline

uv sync --python 3.11 --no-group current --extra compatible
uv run --python 3.11 --no-group current --extra compatible libero-pro smoke

compatible deliberately pins MuJoCo 2.3.7, NumPy 1.26.4, and OpenCV <4.12. Use Python 3.10/3.11: the old MuJoCo release has no suitable prebuilt wheels for newer Python versions. These are a measured reference baseline, not a claim that arbitrary historical environments or different machines produce identical trajectories.

Keep --no-group current --extra compatible on subsequent uv run commands; otherwise uv restores the default dependency selection. To retain a separate baseline environment:

UV_PROJECT_ENVIRONMENT=.venv-compatible uv sync --python 3.11 --no-group current --extra compatible
UV_PROJECT_ENVIRONMENT=.venv-compatible uv run --python 3.11 --no-group current --extra compatible libero-pro smoke

To switch back to the default interpreter and dependencies, use uv sync --python "$(cat .python-version)".

Optional tools

Extra	Dependencies	API and purpose
compatible	Fixed reference simulation dependencies	Compare against the measured upstream baseline
datasets	h5py	libero.libero.utils.dataset_utils.get_dataset_info: inspect downloaded HDF5 demonstration files
video	imageio, imageio-ffmpeg	libero.libero.utils.video_utils.VideoWriter: explicitly collect frames and encode MP4 videos

uv sync --extra datasets --extra video

Installing datasets does not download demonstrations or add a policy training framework. It is unnecessary for task BDDL files and NPZ initial states. Installing video does not enable automatic recording: the caller must create a writer with save_video=True, append frames, and call save() or exit its context manager. The caller may also save observation images with their own video library. See complete API examples.

The default local dependency group current supplies newer simulation constraints to uv's universal lockfile. It adds no new package names and is mutually exclusive with compatible. Published library requirements allow both dependency stacks; the caller's project controls its own resolution. robosuite and BDDL remain pinned to preserve their original APIs. The package has no notebooks; install Jupyter in your own project when needed.

Python API

Existing VLA evaluation code can retain the original benchmark imports:

from pathlib import Path
import numpy as np
from libero.libero import benchmark, get_libero_path
from libero.libero.envs import OffScreenRenderEnv

suite = benchmark.get_benchmark("libero_spatial")()
task = suite.get_task(0)
states = suite.get_task_init_states(0)  # ndarray with the original dtype and shape
bddl = Path(get_libero_path("bddl_files")) / task.problem_folder / task.bddl_file

env = OffScreenRenderEnv(bddl_file_name=str(bddl), camera_heights=128, camera_widths=128)
try:
    env.seed(123)
    env.reset()
    obs = env.set_init_state(states[0])
    obs, reward, done, info = env.step(np.zeros(env.env.action_dim))
    success = env.check_success()
finally:
    env.close()

The alternate entry points from libero_pro_benchmark import benchmark, get_libero_path and from libero_pro_benchmark.envs import OffScreenRenderEnv refer to the same benchmark module and environment class. Original observation keys, 7-dimensional Panda actions, and the four-item step() result are preserved. SegmentationRenderEnv, DummyVectorEnv, and SubprocVectorEnv also retain their original import paths.

Distribution and import names

The installable distribution is named libero-pro-benchmark. It provides three top-level Python import names:

• libero: the original benchmark namespace, including libero.libero.
• libero_pro_benchmark: an additional entry point and package-specific tools.
• perturbation: the original top-level perturbation compatibility module.

project.import-names declares these names to packaging tools; actual modules implement the imports. It does not create aliases by itself, choose a device, or install multiple distributions.

Remove the upstream distribution from the caller's dependencies before adding this package. Install the main package from PyPI; all five required resource packages are installed automatically:

pip install libero-pro-benchmark
# Or, in a uv project:
uv add libero-pro-benchmark

For checkout development, run uv sync in this repository. To test built distributions in another project before uploading them, build all six distributions and supply the local resource wheels:

uv remove libero  # Only if the caller's project declares the upstream dependency.
uv add --find-links /absolute/path/to/LIBERO-PRO-Benchmark/dist/release \
  /absolute/path/to/LIBERO-PRO-Benchmark/dist/release/libero_pro_benchmark-0.1.0-py3-none-any.whl

On import, this package rejects installed libero, libero-pro, or libero-ood distributions, and other distributions declaring an overlapping Import-Name. Its own import metadata helps supporting installers detect overlaps. The upstream package lacks that metadata and installer behavior varies, so installation-time mutual exclusion cannot be guaranteed. PyTorch used by an external policy is allowed.

The optional libero-pro console script lists tasks, audits resources, and runs simulation smoke checks. VLA evaluation can use the Python API directly.

Initial states and configuration

Task metadata retains the original .pruned_init filename; packaged data is stored as .pruned_init.npz. Read it through suite.get_task_init_states() or libero_pro_benchmark.states.load_initial_states(). External code calling torch.load(path) directly must change its loading call.

Default task and initial-state paths are resolved from the main package; get_libero_path("assets") points to the complete installed resource tree. There are no import-time prompts, configuration writes, resource copies, or runtime downloads. Explicit setters use ~/.config/libero-pro-benchmark; the old implicit ~/.libero configuration is not read. Set LIBERO_CONFIG_PATH=/your/config-dir to use a config.yaml containing resource path overrides. Initial-state overrides must point to NumPy-format data. LIBERO_DATASETS_PATH separately overrides the demonstration directory.

Perturbation and state generation

The five upstream perturbation types are preserved in libero_pro_benchmark.perturbation, with the original import perturbation entry point. Both expose the same class and function objects. Bundled YAML and templates are available through importlib.resources.files("libero_pro_benchmark"). See perturbation examples and equivalence limits.

uv run python -m libero_pro_benchmark.generate_init_states \
  --bddl_base_dir /path/to/bddl \
  --output_dir /path/to/init_files --num_inits 50

The generator retains the upstream sampling process and writes NPZ arrays. Resampled states are not substitutes for official evaluation states. The upstream ood_object.yaml has an invalid empty key; this package repairs it to wooden_cabinet and retains the original configuration as a test fixture. Upstream comments, docstrings, and diagnostics are retained as supplied.

Compatibility scope

The preserved benchmark definitions and signatures cover 58 modules, 85 module-level functions, and 271 classes, including their methods. Tests also compare task metadata/order and exercise original path-helper keywords such as get_path_dict(root_location=...) and utils.get_libero_path(key=...).

Compatibility covers the benchmark Python API. NPZ storage, configuration location, resource package location, noninteractive imports, and removal of the training subsystem are explicit differences. Use get_libero_path("assets") instead of constructing paths from module __file__ values. The original repository's training code and maintenance commands are outside this package. Matching APIs do not imply identical physics across dependency versions.

Data integrity and resource coverage

• All 848 initial-state arrays (688 upstream files and 160 official additions) were converted losslessly. Their dtype, shape, count, ordering, and C-order byte SHA256 were checked. No float32 conversion is performed. The original torch.save() files contained NumPy arrays, not Tensor storage.
• Runtime loading uses np.load(..., allow_pickle=False). The offline converter supports the original NumPy archives without PyTorch. Its explicit --allow-torch option is for a separate migration environment containing real dense tensors; unsupported dtypes fail rather than being cast.
• Original assets and BDDL retain their source-content fingerprints. Upstream comments and diagnostics are retained; benchmark code includes the documented storage, configuration, optional-import, and simulator API adaptations. Success predicates and controller formulas are preserved.
• Default dependencies allow physics and rendering differences from the old baseline. Compatibility-mode bitwise equality has been measured for specific samples; it is not guaranteed for all trajectories, machines, or historical environments.

Of 960 registered task entries, 612 have BDDL and nonempty initial states; 409 also have the known required model assets. All entries in 33 suites have resources, and the first task of each suite passed reset, state restoration, and stepping. Remaining gaps include missing BDDL/states, empty state files, and upstream models red_sticker, blue_red_sticker, red_box, and libero_mug_green. Models available in the upstream notebooks, including libero_mug_yellow, are packaged. Resource availability is not a policy success-rate certification.

uv run libero-pro audit --output docs/reports/runs/resource-audit.json
uv run libero-pro audit --strict  # Returns nonzero while upstream resource gaps remain.

Official additions come from LIBERO-Pro on Hugging Face, pinned to revision c86fc3b8293185a6f373677018ff3e37f8391602. Their paths and hashes are recorded in official resource provenance.

macOS and Linux

Native macOS OpenGL offscreen rendering has been tested on Apple Silicon in both dependency modes. Do not set MUJOCO_GL=egl on macOS. Interactive MuJoCo viewers have separate mjpython requirements.

On Linux with EGL and a suitable GPU driver:

uv sync
MUJOCO_GL=egl uv run libero-pro smoke

For CPU software rendering with system OSMesa libraries:

MUJOCO_GL=osmesa uv run libero-pro smoke

The system must supply the selected OpenGL backend and drivers. Simulation rendering and policy inference devices are independent settings. Linux awaits testing on a real machine; no CI or scheduled automation has been added. See manual validation commands.

One upstream scene XML filename contained a DEL control character rejected by current archive tools. The packaged filename is normalized to yellow_book.xml, with unchanged contents and an explicit path mapping. The original object class uses a different model directory.

Validation and publication

docs/reports/README.md summarizes the principal findings and limitations. Git tracks compact reports, provenance, original API/task fixtures, tests, reproduction tools, and uv.lock. Detailed local outputs belong in ignored docs/reports/runs/; regression tests do not depend on them. Exact versions in the reports are historical measurements and must remain attached to their results after dependency upgrades.

uv sync --group dev
uv run --group dev pytest -q
uv run python tools/build_packages.py

The main distribution contains benchmark code, task BDDL files, initial-state arrays, configurations, and templates. Five mandatory libero-pro-benchmark-assets-1 through -5 distributions contain the original XML, meshes, and textures. A single uv sync installs all of them; no resource extra or first-use download is needed. The resource packages use non-editable local path dependencies, while the main package remains editable.

The former single wheel was about 345 MiB. The split distributes the same resource bytes across five smaller wheels and leaves the main wheel about 5 MiB. Every wheel and source archive must pass the 100,000,000-byte release check before publication. Run uv build to build only the main distribution; use tools/build_packages.py to build all six. Publish the resource distributions before the main distribution. Local reports are excluded from source archives and Git. See package layout, installation, and manual release checks and the validated split results.

Attribution and license

This is an independent extraction, not an official release. Original code retains the MIT license; the upstream dataset is labeled CC BY 4.0. Upstream asset ownership and licensing continue to apply. Cite LIBERO and LIBERO-PRO when using the benchmark in research.