iPLAN 0.1.3

Codebase for iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning

This repository is the codebase for our paper.

iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning

This repository was originally forked from https://github.com/oxwhirl/pymarl and https://github.com/carolinewang01/dm2. The MAPPO baseline comes from https://github.com/uoe-agents/epymarl.

Table of Contents

• Dependencies
• Installation
• Running iPLAN
• Ablation Study
  • IPPO, IPPO-BM, IPPO-GAT, iPLAN-Hard, iPLAN-FC
• Baselines
• Helper Functions
  • Compute Navigation Metrics
  • Generate Animation
  • Plot Reward Curve
• Results
• Animation
• Citation

Dependencies

• PyTorch (1.13.1 + cu116) (GPU)
• stable-baselines3
• Heterogeneous_Highway_Env (Forked from highway-env)
• scared
• PyYAML

Note: Please our modified Highway-env given in Heterogeneous_Highway_Env as there are major changes from the initial version of highway-env. Also, Multi-agent Particles used in our repo are different. Please use the code given in envs/mpe folder.

Installation

First, install dependencies

pip install stable-baselines3[extra] pyyaml sacred

Then install our forked version of Highway-env

pip install Heterogeneous_Highway_Env

Finally, install iPLAN package

pip install iPLAN

Running iPLAN

In the configuration file config/default.yaml, set up environments needed for your experiment:

• Set environment env: MPE for Non-cooperative Navigation and highway for Heterogeneous Highway
• Set difficulty level difficulty:
  • easy for easy (Non-cooperative Navigation) or mild (Heterogeneous Highway) scenario.
  • hard for hard (Non-cooperative Navigation) or chaotic (Heterogeneous Highway) scenario.
• Set Behavior_enable: True
• Set GAT_enable: True and GAT_use_behavior: True
• Set soft_update_enable: True and behavior_fully_connected: False
• Run python3 main.py

Results, including printed logs, saved models and tensorboard logger, are stored in the folder results

Ablation Study

When running experiments for ablation study, please only change the hyperparameters mentioned in the configuration file config/default.yaml and keep those the same as they are in the iPLAN experiment.

Running IPPO

• Set Behavior_enable: False
• Set GAT_enable: False and GAT_use_behavior: False
• Run python3 main.py

Running IPPO-BM

• Set Behavior_enable: True
• Set GAT_enable: False and GAT_use_behavior: False
• Run python3 main.py

Running IPPO-GAT

• Set Behavior_enable: False
• Set GAT_enable: True and GAT_use_behavior: True
• Run python3 main.py

Running iPLAN-Hard

• Set soft_update_enable: False
• Run python3 main.py

Running iPLAN-FC

• Set behavior_fully_connected: True
• Run python3 main.py

Baselines

Baselines used in this paper could be found in the baselinesfolder, where the organization of files is similar to the main directory of iPLAN. Please change the environment setting in config/default.yaml before experiments. No extra changes need.

• QMIX: Run python3 baselines/QMIX/main.py
• MAPPO: Run python3 baselines/MAPPO/main.py

Helper Functions

Notebooks for helper function are given in helper folder. Please follow the instructions below:

Compute Navigation Metrics

(Only for Heterogeneous Highway) In the configuration file config/default.yaml

• Set metrics_enable: True
• Set num_test_episodes larger than batch_size_run
• Run python3 main.py

Then you will get printed navigation metrics after the execution logs of each episode.

To compute the navigation metrics, use the notebook helper/RL_results_metrics.ipynb to compute averaged navigation metrics from the printed log file (usually given in results/sacred).

Generate Animation

(Only for Heterogeneous Highway) In the configuration file config/default.yaml

• Set animation_enable: True
• (Recommended) Set metrics_enable: True, Set num_test_episodes larger than batch_size_run
• Run main.py

Screenshots of the Heterogeneous Highway are stored in the animation folder. Use the notebook helper/Gif_helper.ipynb to generate animation from screenshots.

Plot Reward Curve

The printed log file are usually given in results/sacred.

• Choose the log file you want to recover, use the notebook helper/RL_results_repack.ipynb to convert the log file into .csv file.
• Use the notebook RL Visualization Helper - Highway.ipynb (Heterogeneous Highway) or RL Visualization Helper - MPE.ipynb (Non-cooperative Navigation) to plot the reward curve from the generated .csv files for each approaches and scenarios.

Citation

@article{wu2023iplan,
  title={iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning},
  author={Wu, Xiyang and Chandra, Rohan and Guan, Tianrui and Bedi, Amrit Singh and Manocha, Dinesh},
  journal={arXiv preprint arXiv:2306.06236},
  year={2023}
}