morl-baselines 1.3.0

Implementations of multi-objective reinforcement learning (MORL) algorithms.

MORL-Baselines

MORL-Baselines is a library of Multi-Objective Reinforcement Learning (MORL) algorithms. This repository aims to contain reliable MORL algorithms implementations in PyTorch.

It strictly follows MO-Gymnasium API, which differs from the standard Gymnasium API only in that the environment returns a numpy array as the reward.

For details on multi-objective MDPs (MOMDPs) and other MORL definitions, we suggest reading A practical guide to multi-objective reinforcement learning and planning. An overview of some techniques used in various MORL algorithms is also provided in Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework.

A tutorial on MO-Gymnasium and MORL-Baselines is also available:

Features

• Single and multi-policy algorithms under both SER and ESR criteria are implemented.
• All algorithms follow the MO-Gymnasium API.
• Performances are automatically reported in Weights and Biases dashboards.
• Linting and formatting are enforced by pre-commit hooks.
• Code is well documented.
• All algorithms are automatically tested.
• Utility functions are provided e.g. pareto pruning, experience buffers, etc.
• Performances have been tested and reported in a reproducible manner.
• Hyperparameter optimization available.

Implemented Algorithms

Name	Single/Multi-policy	ESR/SER	Observation space	Action space	Paper
GPI-LS + GPI-PD	Multi	SER	Continuous	Discrete / Continuous	Paper and Supplementary Materials
GPI-LS (Jax)	Multi	SER	Continuous	Discrete / Continuous	Paper and Supplementary Materials
MORL/D	Multi	SER / ESR	Discrete / Continuous	Discrete / Continuous	Paper
Envelope Q-Learning	Multi	SER	Continuous	Discrete	Paper
CAPQL	Multi	SER	Continuous	Continuous	Paper
PGMORL	Multi	SER	Continuous	Continuous	Paper / Supplementary Materials
Pareto Conditioned Networks (PCN)	Multi	SER/ESR 1	Continuous	Discrete / Continuous	Paper
Lorenz Conditioned Networks (LCN)	Multi	SER/ESR 1	Continuous	Discrete / Continuous	Paper
Pareto Q-Learning	Multi	SER	Discrete	Discrete	Paper
MO Q learning	Single	SER	Discrete	Discrete	Paper
MPMOQLearning (outer loop MOQL)	Multi	SER	Discrete	Discrete	Paper
Optimistic Linear Support (OLS)	Multi	SER	/	/	Section 3.3 of the thesis
Expected Utility Policy Gradient (EUPG)	Single	ESR	Discrete	Discrete	Paper
Iterated Pareto Referent Optimisation (IPRO)	Multi	SER	Continuous	Discrete	Paper
Iterated Pareto Referent Optimisation-2D (IPRO-2D) 2	Multi	SER	Continuous	Discrete	Paper
Non-Linear Multi-Objective PPO (NLMOPPO)	Single	SER	Continuous	Discrete	Derivation of the policy-gradient update in this paper
:warning: Some of the algorithms have limited features.

1: PCN assumes environments with deterministic transitions.
2: IPRO-2D is a two-objective specialisation of IPRO designed for efficiency gains, but it cannot be applied to settings with more than two objectives.

Benchmarking

MORL-Baselines participates to Open RL Benchmark which contains tracked experiments from popular RL libraries such as cleanRL and Stable Baselines 3.

We have run experiments of our algorithms on various environments from MO-Gymnasium. The results can be found here: https://wandb.ai/openrlbenchmark/MORL-Baselines. An issue tracking all the settings is available at #43. Some design documentation for the experimentation protocol are also available on our Documentation website.

An example visualization of our dashboards with Pareto support is shown below:

Structure

As much as possible, this repo tries to follow the single-file implementation rule for all algorithms. The repo's structure is as follows:

• examples/ contains a set of examples to use MORL Baselines with MO-Gymnasium environments.
• common/ contains the implementation recurring concepts: replay buffers, neural nets, etc. See the documentation for more details.
• multi_policy/ contains the implementations of multi-policy algorithms.
• single_policy/ contains the implementations of single-policy algorithms (ESR and SER).

Citing the Project

If you use MORL-Baselines in your research, please cite our NeurIPS 2023 paper:

@inproceedings{felten_toolkit_2023,
	author = {Felten, Florian and Alegre, Lucas N. and Now{\'e}, Ann and Bazzan, Ana L. C. and Talbi, El Ghazali and Danoy, Gr{\'e}goire and Silva, Bruno Castro da},
	title = {A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning},
	booktitle = {Proceedings of the 37th Conference on Neural Information Processing Systems ({NeurIPS} 2023)},
	year = {2023}
}

Maintainers

MORL-Baselines is currently maintained by Florian Felten (@ffelten) and Lucas N. Alegre (@LucasAlegre).

Contributing

This repository is open to contributions and we are always happy to receive new algorithms, bug fixes, or features. If you want to contribute, you can join our Discord server and discuss your ideas with us. You can also open an issue or a pull request directly.

Acknowledgements

• Willem Röpke (@wilrop), for his original implementation of IPRO(-2D) and reimplementation of non-linear MO-PPO and Pareto Q-Learning.
• Mathieu Reymond, for providing us with the original implementation of PCN.
• Denis Steckelmacher and Conor F. Hayes, for providing us with the original implementation of EUPG.
• Jayden Teoh, for providing several improvements in different algorithms and the implementation of MOSAC discrete.