MAGMA-BENCH

Evaluate agents on interactive robotic tasks with MAGMA. MAGMA-BENCH loads compiled benchmark episodes, runs them in simulation, communicates with an agent through the MAGMA HTTP protocol, and saves detailed results and metrics.

Version 2.0.1 adds versioned benchmark downloads. Dataset releases have independent versions; MAGMA-BENCH dataset 1.0.0 requires MAGMA Bench 2.0.1, Core 2.0.1, and Scenarios 2.1.1 exactly.

Official documentation · Report an issue

Installation

Python 3.12 is required. Simulation installation has been checked on Linux x86_64. GPU simulation and rendering require compatible system drivers.

python -m pip install "magma_bench==2.0.1"

This installs magma_core[simulation]>=2.0.0,<3.0.0 and magma_scenarios>=2.1.0,<3.0.0 with their Python dependencies. The benchmark files and the agent server are separate inputs and are not bundled with this package.

To upgrade to the latest stable release, use:

python -m pip install --upgrade magma_bench

Usage

Download the latest stable official dataset, or request an exact release:

magma-bench fetch
magma-bench fetch --version 1.0.0

Releases come from lbernat/MAGMA-BENCH. The default destination is $XDG_CACHE_HOME/magma-bench/datasets/v1.0.0/, or ~/.cache/magma-bench/datasets/v1.0.0/ when XDG_CACHE_HOME is unset. Each download records its immutable Hub commit and file checksums. A published tag must never be moved. Repeating a fetch reuses a complete installation; a changed tag or damaged installation is rejected. To recover, choose a new empty --path or remove the damaged cache directory before fetching again.

Start a compatible MAGMA agent server, then run the downloaded release offline:

magma-bench run \
  --dataset 1.0.0 \
  --results-path ./eval/experiment-1 \
  --agent-address http://127.0.0.1:8888 \
  --run-name experiment-1

run never downloads data. --dataset accepts an exact version, with or without v, and looks only in the conventional cache. Package versions are checked before runtime initialization; a mismatch prints the exact installation command. Different dataset versions can coexist on disk, but running them may require separate Python environments for their different package pins.

To download to a custom destination, use:

magma-bench fetch --version 1.0.0 --path ./compiled-benchmark

Run that directory, or your own compiled benchmark, with --benchmark-root:

magma-bench run \
  --benchmark-root /path/to/compiled-benchmark \
  --results-path /path/to/results/experiment-1 \
  --agent-address http://127.0.0.1:8888 \
  --run-name experiment-1

--dataset and --benchmark-root are mutually exclusive; one is required. Every benchmark directory, including a custom build, must contain both benchmark.json and a release manifest.json:

{
  "schema_version": "1.0",
  "benchmark_version": "1.0.0",
  "requirements": {
    "magma_bench": "2.0.1",
    "magma_core": "2.0.1",
    "magma_scenarios": "2.1.1"
  }
}

Use the same benchmark_version in both files and exact package versions in requirements. The compiled artifact schema remains 1.8; the distribution manifest has its own schema 1.0. Install the official release environment with:

python -m pip install "magma_bench==2.0.1" "magma_core[simulation]==2.0.1" "magma_scenarios==2.1.1"

Add the manifest to older custom builds before using this CLI. Its contents participate in result fingerprints, so changing pins requires a new result directory. Old draft evaluations are development results; start fresh for the published dataset 1.0.0. stats checks manifest consistency without requiring the pinned runtime packages.

The agent must expose /health, /v1/info, and /v1/responses using protocol version 2.0. Configuration can be supplied with --config-path; command-line options override its benchmark settings.

Useful options include:

magma-bench run --help
magma-bench run --benchmark-root /path/to/benchmark --results-path ./eval/coffee --scenarios coffee_comp
magma-bench run --benchmark-root /path/to/benchmark --results-path ./eval/no-judge --skip-judge
magma-bench run --benchmark-root /path/to/benchmark --results-path ./eval/logged --model-logs --videos all
magma-bench run --benchmark-root /path/to/benchmark --results-path ./eval/failures --videos planner-failure

Video output uses the optional dependencies:

python -m pip install "magma_bench[video]==2.0.1"

Merge all completed parts of one evaluation into a YAML analysis:

magma-bench stats ./stats.yaml ./eval/model-name /path/to/compiled-benchmark

The results directory must contain one subdirectory per part, each with run.json and result.json. All compiled episodes must be covered exactly once. Partial runs and infrastructure failures stop the command without writing the YAML. The name field is the results directory name. The file includes success rates, conditional robustness, failure statuses, and scenario, skeleton, semantic, length, and track breakdowns. Stage progression is the mean of each episode's fraction of completed stages, using stage counts from the compiled benchmark. A benchmark fingerprint mismatch is reported in the YAML and on stderr because episode IDs alone cannot certify that compiled contents match exactly.

Version 2 uses result schema 3.0. Terminal statuses are success, invalid_format, textual_hallucination, non_requested_action, non_physical_exceeded, judge_failure, env_failure, budget_exceeded, and infrastructure_failure. An infrastructure failure returns the episode to the end of its current simulator group, for at most three attempts per launch. The progress log records every attempt; episode results, model logs, and videos retain the final attempt. Resume a compatible run with --results-path; a resumed infrastructure failure receives a fresh retry allowance. Result schema 2.0 and older agent adapters are unsupported.

See the official documentation for benchmark preparation, configuration, agent setup, metrics, and result formats.

Install from source

Install the tagged release from GitHub:

python -m pip install "magma_bench @ git+https://github.com/MAGMA-rob/magma-bench.git@v2.0.1"

Dependencies are resolved from PyPI. For local development, clone the repository and run python -m pip install -e ".[dev,video]".

License

BSD 2-Clause.