robot-eval-logger 0.0.1

A flexible logger for robot evaluations with visualization and data storage capabilities

robot_eval_logger

This package provides a logger for robot evaluations. The main functionalities are:

• Flexible logging: with a few lines of addition to your evaluation code, you can log any data you want
• Visualization: visualize the logged data (e.g. videos, success rates) easily through wandb (weights & biases).
• Data Storage: store the logged data locally and / or upload to HuggingFace datasets

Installation

Install the package from source with

git clone [e-mail]:zhouzypaul/robot_eval_logger.git
cd robot_eval_logger
pip install -e .

Getting Started

First, create the EvalLogger instance. When creating the EvalLogger, you can optionally pass in three functionalities this package provides:

• wandb_logger: log metrics to wandb. Here is an example of the logged metrics.
• frames_visualizer: make videos, film strips, and success information plotted against initial and final frames to visualize the evaluation. See here for an example.
• data_saver: save the logged data locally and / or upload to HuggingFace datasets If you don't need some / all of these, pass in None for the corresponding argument.

from robot_eval_logger import (
    EvalLogger,
    FrameVisualizer,
    HuggingFaceStorage,
    LocalStorage,
    WandBLogger,
)

# wandb logger that logs metrics to wandb
wandb_config = WandBLogger.get_default_config()
wandb_logger = WandBLogger(
    wandb_config=wandb_config,
    variant={"exp_name": "testing_eval_logger"},
)

# frames visualizer that makes videos, film strips, and success information plotted against initial and final frames to visualize the evaluation
frames_visualizer = FrameVisualizer(
    episode_viz_frame_interval=10,  # when creating a film strip of the eval trajectory, visualize a frame every 10 frames
    success_viz_every_n=3,  # for every 3 episodes, visualize success information alongside initial and final frames
    periodic_log_initial_and_final_frames=True,  # visualize initial and final frames alongside success information
)

# data saver
# Option 1: save data locally
data_saver = LocalStorage(
    storage_dir='path/to/your/storage',
)
# Option 2: save data to HuggingFace
data_saver = HuggingFaceStorage(
    storage_dir='path/to/your/storage',
    repo_id="HF_USERNAME/eval_logger",
    hf_dir_name="eval_data",
)

# create the eval logger
# You can pass in None for any of the arguments if you don't need some / all of the functionalities
eval_logger = EvalLogger(
    wandb_logger=wandb_logger,
    frames_visualizer=frames_visualizer,
    data_saver=data_saver,
)

You can save the metadata for the evaluation:

eval_logger.save_metadata(
    location="berkeley",
    robot_name="widowx_dummy",
    robot_type="widowx",
    evaluator_name="dummy_tester",
    eval_name="dummy_eval",
)

After each evaluation episode, log the episode data. You need to pass in the episode number, the prefix for the logging (e.g. language prompt), the success status, and the frames to log. You can also pass in any additional kwargs to log any data you want during an evaluation.

kwargs = {}  # put any additional kwargs here
eval_logger.log_episode(
    i_episode=i_episode,
    logging_prefix="put the mushroom into the pot",
    episode_success=True,
    frames_to_log=[obs["image"] for obs in obs_list],
    **kwargs,
)

You can also log the evaluation throughput in terms of the number of evaluation steps taken per minute:

eval_logger.log_step()

Example Usage

See examples/manipulator_eval.py for an example of using the EvalLogger class in an evaluation loop on WidwoX robots. (For now, this script runs null actions on the robot, and you can use it to test the logger functionalities.)

python examples/manipulator_eval.py --ip <robot_ip>

Uploading data to HF

You must be authenticated with a write token with

huggingface-cli login

Contributing

To enable code checks and auto-formatting, please install pre-commit hooks (run this in the root directory):

pre-commit install

The hooks should now run before every commit. If files are modified during the checks, you'll need to re-stage them and commit again.