DTR-Bench 0.1.0

DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime

DTR-Bench

---

An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime

📝 Table of Contents

• About
• Getting Started
• Module Description
• Usage
• Reference
• Special Thanks
• Acknowledge

🧐 About

DTR-Bench (DTR-Bench) is an expanding reinforcement learning simulation platform with a unified pipeline including hyperparameter search, training, evaluation, and visualisation.

🏁 Getting Started

These instructions will get you a copy of the project up and running on your local machine.

Prerequisites

• Python 3.10: The project is developed using Python 3.10. It is recommended to use the same version to avoid compatibility issues.

Installing

1. Install the DTR-Gym and the DTR-Bench

pip install dtrgym

2. Install the required packages

cd SimMedEnv
pip install -r requirements.txt

3. Test the installation

python test_installation.py

Running the getting started demo

We provide a simple example to show how to use the DTR-Bench. The example is in the get_start.py file.

You can run the example by:

python get_start.py

After running the example, you will see the following a plot like this. It shows the effect of the treatment given by the train RL policy in the simulation environment.

🎈 Module Description

Offpolicy RL Algorithms

The DTRBench provides a set of off-policy RL algorithms to train the RL policy in the simulation environments. These policies are developed based on Tianshou. The off-policy RL algorithms include:

Discrete	RNN-based	Continuous
DDQN	DQN-rnn	DDPG
DDQN-dueling	DDQN-rnn	TD3
DQN	C51-rnn	SAC
C51	discrete-SAC-rnn
discrete-SAC

Run RL Functions

Running functions in reinforcement learning, which includes hyperparameters grid-searching, policies training and evaluation, and baseline policies evaluation.

Visualisation

The DTRBench provides a standard visualisation tool to visualise the treatment effect of the trained RL policy in the simulation environments. It enables visualising the environment states, observations, actions, and the reward.

🔧 Usage

The DTR-Bench provides a set of APIs to interact with the simulation environments and the benchmark policies.

It can be used to:

1. Create a simulation environment.
2. Optimize the hyperparameters of the RL policy training.
3. Train a RL policy in the simulation environments.
4. Visualize the treatment effect of the trained RL policy.

Create a Simulation Environment

import gymnasium as gym
import DTRGym  # this line is necessary!

gym.make('AhnChemoEnv-discrete', n_act=11)

Please remember to import DTRGym to register the simulation environments.

Hyperparameters Search

1. Discrete policies todo: change the cd path

cd SimMedEnv
export PYTHONPATH="."

python DTRBench/run_rl/online_discrete_search.py --policy_name=DQN --task SimGlucoseEnv --n_trials 100 --num_actions 11 --setting 1

2. continuous policies

cd SimMedEnv
export PYTHONPATH="."

python DTRBench/run_rl/online_continuous_search.py --policy_name=DDPG --task OberstSepsisEnv --n_trials 100 --setting 1

Online Training and Testing

1. Discrete Policies

cd SimMedEnv
export PYTHONPATH="."
conda activate torch

python DTRBench/run_rl/online_discrete_retrain.py --policy_name=DQN --task SimGlucoseEnv --num_actions 11 --setting 1

2. Continuous Policies

cd SimMedEnv
export PYTHONPATH="."
conda activate torch

python DTRBench/run_rl/online_continuous_retrain.py --policy_name=DDPG --task SimGlucoseEnv --setting 1

3. Baseline Policies (RandomPolicy, MaxPolicy, MinPolicy)

cd SimMedEnv
export PYTHONPATH="."
conda activate torch

python DTRBench/run_rl/online_baseline.py --task OberstSepsisEnv

Visualisation

cd SimMedEnv
export PYTHONPATH="."

python DTRBench/visual_fn/visual.py

Download the Hyperparameters and Test Results

The hyperparameters and test results are stored on Kaggle.

Reference

If you use the DTR-Bench in your research, please cite the following paper:

to be updated

✍️ Sepcial Thanks

Special thanks to the following contributors that make the DTR-Bench possible:

• @Mingcheng Zhu - who developed DTRGym and produced extensive DTRBench experiments.
• To be continued

🎉 Acknowledgement

• Tianshou
• Gymnasium
• Optuna