No project description provided
Project description
Embodied Agent Interface (EAI): Benchmarking LLMs for Embodied Decision Making
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu
Stanford Vision and Learning Lab, Stanford University
Dataset Highlights
- Standardized goal specifications.
- Standardized modules and interfaces.
- Broad coverage of evaluation and fine-grained metrics.
Overview
We aim to evaluate Large Language Models (LLMs) for embodied decision-making. While many works leverage LLMs for decision-making in embodied environments, a systematic understanding of their performance is still lacking. These models are applied in different domains, for various purposes, and with diverse inputs and outputs. Current evaluations tend to rely on final success rates alone, making it difficult to pinpoint where LLMs fall short and how to leverage them effectively in embodied AI systems.
To address this gap, we propose the Embodied Agent Interface (EAI), which unifies:
- A broad set of embodied decision-making tasks involving both state and temporally extended goals.
- Four commonly used LLM-based modules: goal interpretation, subgoal decomposition, action sequencing, and transition modeling.
- Fine-grained evaluation metrics, identifying errors such as hallucinations, affordance issues, and planning mistakes.
Our benchmark provides a comprehensive assessment of LLM performance across different subtasks, identifying their strengths and weaknesses in embodied decision-making contexts.
Installation
-
Create and Activate a Conda Environment:
conda create -n eai-eval python=3.8 -y conda activate eai-eval
-
Install
eai
:You can install it from pip:
pip install eai-eval
Or, install from source:
git clone https://github.com/embodied-agent-interface/embodied-agent-interface.git cd embodied-agent-interface pip install -e .
-
(Optional) Install iGibson for behavior evaluation:
If you need to use
behavior_eval
, install iGibson. Follow these steps to minimize installation issues:-
Make sure you are using Python 3.8 and meet the minimum system requirements in the iGibson installation guide.
-
Install CMake using Conda (do not use pip):
conda install cmake
-
Install
iGibson
: We provide an installation script:python -m behavior_eval.utils.install_igibson_utils
Alternatively, install it manually:
git clone https://github.com/embodied-agent-interface/iGibson.git --recursive cd iGibson pip install -e .
-
Download assets:
python -m behavior_eval.utils.download_utils
We have successfully tested installation on Linux, Windows 10+, and macOS.
-
Quick Start
-
Arguments:
eai-eval \ --dataset {virtualhome,behavior} \ --mode {generate_prompts,evaluate_results} \ --eval-type {action_sequencing,transition_modeling,goal_interpretation,subgoal_decomposition} \ --llm-response-path <path_to_responses> \ --output-dir <output_directory> \ --num-workers <number_of_workers>
Run the following command for further information:
eai-eval --help
-
Examples:
-
Evaluate Results
Make sure to download our results first if you don't want to specify <path_to_responses>
python -m eai_eval.utils.download_utils
Then, run the commands below:
eai-eval --dataset virtualhome --eval-type action_sequencing --mode evaluate_results eai-eval --dataset virtualhome --eval-type transition_modeling --mode evaluate_results eai-eval --dataset virtualhome --eval-type goal_interpretation --mode evaluate_results eai-eval --dataset virtualhome --eval-type subgoal_decomposition --mode evaluate_results eai-eval --dataset behavior --eval-type action_sequencing --mode evaluate_results eai-eval --dataset behavior --eval-type transition_modeling --mode evaluate_results eai-eval --dataset behavior --eval-type goal_interpretation --mode evaluate_results eai-eval --dataset behavior --eval-type subgoal_decomposition --mode evaluate_results
-
Generate Pormpts
To generate prompts, you can run:
eai-eval --dataset virtualhome --eval-type action_sequencing --mode generate_prompts eai-eval --dataset virtualhome --eval-type transition_modeling --mode generate_prompts eai-eval --dataset virtualhome --eval-type goal_interpretation --mode generate_prompts eai-eval --dataset virtualhome --eval-type subgoal_decomposition --mode generate_prompts eai-eval --dataset behavior --eval-type action_sequencing --mode generate_prompts eai-eval --dataset behavior --eval-type transition_modeling --mode generate_prompts eai-eval --dataset behavior --eval-type goal_interpretation --mode generate_prompts eai-eval --dataset behavior --eval-type subgoal_decomposition --mode generate_prompts
Docker
We provide a ready-to-use Docker image for easy installation and usage.
First, pull the Docker image from Docker Hub:
docker pull jameskrw/eai-eval
Next, run the Docker container interactively:
docker run -it jameskrw/eai-eval
Test docker
eai-eval
By default, this will start generating prompts for goal interpretation in Behavior.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.