MEAL-Bench 0.1.0

MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning

MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning

MEAL is the first Continual Multi-Agent Reinforcement Learning (CMARL) benchmark, using the Overcooked environment, implemented in JAX for high-performance training and evaluation. The project focuses on enabling agents to continuously learn and adapt to new environments without forgetting previously acquired skills.

Key Features

• JAX-based Implementation: Leverages JAX for efficient, hardware-accelerated training
• Procedurally Generated Environments: An infinite stream of solvable, variable‑sized cooperative challenges
• Visualization: Tools for analyzing environments and training progress
• 3 Difficulty Levels: easy, medium, and hard
• Continual Learning Methods:
  • Elastic Weight Consolidation (EWC)
  • Online EWC
  • Memory Aware Synapses (MAS)
  • L2 Regularization
  • Fine-Tuning (FT)
  • PackNet
  • Averaged Gradient Episodic Memory (AGEM)
• Multi-Agent Algorithms:
  • Independent Proximal Policy Optimization (IPPO)
  • Value Decomposition Networks (VDN)

Installation

MEAL requires Python 3.10 and uses JAX with CUDA support by default. We recommend creating an isolated Python environment first, then installing MEAL with pip.

Create an Environment

Option 1 — venv

python3.10 -m venv .venv
# macOS/Linux
source .venv/bin/activate
# Windows (CMD)
.\.venv\Scripts\activate.bat
# Windows (PowerShell)
.\.venv\Scripts\Activate.ps1

Option 2 — Conda

conda create -n meal python=3.10 -y
conda activate meal

Install MEAL

# From PyPI (stable release)
pip install MEAL-Bench

# Core install
pip install -e .

# Visualization tools
pip install -e ".[viz]"

# Utility tools
pip install -e ".[utils]"

(Optional) GPU acceleration with JAX

pip install -U "jax[cuda12]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
# or for CUDA 11```
pip install -U "jax[cuda11]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

Usage

Running IPPO with Continual Learning

# Run IPPO with Elastic Weight Consolidation (EWC)
python -m baselines.PPO_CL --cl_method EWC

Environment Generation

MEAL includes tools for generating new Overcooked layouts:

# Generate a random environment
python -m jax_marl.environments.overcooked_environment.env_generator --save

# Visualize environment generation steps
python -m jax_marl.environments.overcooked_environment.env_generator --show

# Generate step-by-step visualization images
python -m jax_marl.viz.env_gen_steps --difficulty medium

Example of environment generation steps (generated using jax_marl.viz.env_gen_steps):

Step 1: Initialize Grid	Step 2: Place Interactive Tiles	Step 3: Add Walls	Step 4: Place Agents & Clean Up
Create empty grid with border walls	Add goal, pot, onion pile, and plate pile	Add additional walls to reach target density	Place agents and remove unreachable areas

Environment Difficulty

MEAL generates layouts with varying difficulty levels to test agent adaptability: The following images illustrate the different layouts:

Easy Layout	Medium Layout	Hard Layout

The difficulty of layouts is determined by the following criteria:

Difficulty Level	Grid Width	Grid Height	Obstacle Density
Easy	6-7	6-7	15%
Medium	8-9	8-9	25%
Hard	10-11	10-11	35%

Project Structure

• assets/: Screenshots, GIFs, and other assets
  • gifs/: Example training progress GIFs
  • screenshots/: Example environment layouts and training results
• examples/: Example scripts for using MEAL environments
• experiments/: CMARL experiments on MEAL
  • continual/: Continual learning method implementations
  • evaluation/: Evaluating trained models
  • experimental/: Algorithm/Method prototyping
  • model/: Neural network architectures (MLP, CNN)
  • partner_adaptation/: Continual partner adaptation experiments
  • results/: Scripts for downloading, aggregating, and plotting results
    • download/: Scripts for downloading and storing data
    • plotting/: Plotting scripts for training results
    • numerical/: Numerical results and analysis
• meal/: Core MEAL framework
  • env/: Overcooked environment creation and management
  • visualization/: Visualization tools
  • wrappers/: Environment wrappers
• scripts/: Utility scripts for training
• tests/: Unit tests for various components

Contributing

Contributions to MEAL are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• The Overcooked environment is based on JaxMARL.
• Our experiments were managed using WandB.

Citation

If you use our work in your research, please cite it as follows:

@article{tomilin2025meal,
  title={MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning},
  author={Tomilin, Tristan and van den Boogaard, Luka and Garcin, Samuel and  Ruhdorfer, Constantin and Grooten, Bram and Bulling, Andreas and Pechenizkiy, Mykola and Fang, Meng},
  journal={arXiv preprint arXiv:2406.01234},
  year={2025}
}