offlinerllib 0.0.3

A python module desgined for Offline RL algorithms developing and benchmarking.

OfflineRL-Lib

🚧 This repo is not ready for release, benchmarking is ongoing. 🚧

OfflineRL-Lib provides unofficial and benchmarked PyTorch implementations for selected OfflineRL algorithms, including:

• Extreme Q-Learning
• Implicit Q-Learning
• Decision Transformer
• EDAC
• SAC-N
• TD3-BC

Benchmark Results

When certain design choices, e.g. the choice of autodiff backend (jax or tf or pytorch) vary, the preference for each hyper-parameters may vary as well. Hence when benchmarking, we tested each algorithm's performace in three ways:

• Paper Performance: the performance reported in white paper;
• OfflineRL-Lib (with paper args): the performance obtained by using OfflineRL-Lib implementation and the configs in paper or original implementations;
• OfflineRL-Lib (with CORL args): the performance obtained by using OfflineRL-Lib implementation and the configs in CORL.

For the last option, arguments are directly borrowed from CORL. CORL provides simplified single-file implementations of these algorithms as well as their finetuned hyper-parameters based on pytorch, please check their repo as well.

XQL :page_facing_up: :chart_with_upwards_trend:

Task	Dataset Quality	Paper Performance (consistent)	Paper Performance (tuned)	OfflineRL-Lib (paper args) (consistent)	OfflineRL-Lib (paper args) (tuned)
halfcheetah	random-v2	NA	NA	NA	NA
	medium-v2	47.7	48.3	NA	47.9±0.2
	medium-replay-v2	44.8	45.2	NA	44.3±0.4
	medium-expert-v2	89.8	94.2	NA	92.1±1.0
hopper	random-v2	NA	NA	NA	NA
	medium-v2	71.1	74.2	NA	67.0±6.8
	medium-replay-v2	97.3	100.7	NA	96.9±6.2
	medium-expert-v2	107.1	111.2	NA	101.9±5.2
walker2d	random-v2	NA	NA	NA	NA
	medium-v2	81.5	84.2	NA	83.8±0.4
	medium-replay-v2	75.9	82.2	NA	76.5±5.2
	medium-expert-v2	110.1	112.7	NA	110.1±0.4

IQL :page_facing_up: :chart_with_upwards_trend:

Task	Dataset Quality	Paper Performance	OfflineRL-Lib (with paper args)	OfflineRL-Lib (with CORL args)
halfcheetah	random-v2	NA	9.4±3.9	13.5±3.9
	medium-v2	47.4	47.3±0.2	48.6±0.2
	medium-replay-v2	44.2	43.7±0.7	44.3±0.4
	medium-expert-v2	86.7	89.7±2.9	93.9±1.6
	full-replay-v2	NA	73.5±0.8	74.9±0.3
	expert-v2	NA	94.8±0.4	95.7±2.6
hopper	random-v2	NA	7.9±0.3	7.3±0.1
	medium-v2	66.3	64.8±7.2	54.5±1.6
	medium-replay-v2	94.7	93.4±7.9	41.5±23.0
	medium-expert-v2	91.5	97.8±9.0	108.1±3.1
	full-replay-v2	NA	104.5±6.0	106.3±1.0
	expert-v2	NA	110.1±0.8	103.8±7.9
walker2d	random-v2	NA	6.0±1.0	3.1±0.9
	medium-v2	78.3	83.5±2.2	81.3±8.7
	medium-replay-v2	73.9	66.6±16.2	77.0±7.3
	medium-expert-v2	109.6	108.9±2.5	112.4±0.8
	full-replay-v2	NA	92.9±3.5	99.2±0.7
	expert-v2	NA	109.7±0.3	112.6±0.4