or-gymnasium

or-gymnasium packages a collection of Gymnasium environments for operations research reinforcement learning problems adapted from the original or-gym project. The main changes are to the environment interfaces so they work with the latest Gymnasium API.

Installation

Install the package with pip to use the bundled environment registrations:

pip install or-gymnasium

You can also copy the environment files you need into your own project, register them with Gymnasium, and use them without installing this package.

Quickstart

Importing or-gymnasium registers the environments with Gymnasium.

import gymnasium as gym
import or_gymnasium

env = gym.make("Newsvendor-v0")
observation, info = env.reset(seed=123)
observation, reward, terminated, truncated, info = env.step(env.action_space.sample())

Using Copied Environment Files

After copying an environment file, register the environment with Gymnasium using the copied module path and class name.

import gymnasium as gym

gym.register(
    id="Newsvendor-v0",
    entry_point="my_project.envs.newsvendor:NewsvendorEnv",
)

env = gym.make("Newsvendor-v0")
observation, info = env.reset(seed=123)
observation, reward, terminated, truncated, info = env.step(env.action_space.sample())

Replace my_project.envs.newsvendor:NewsvendorEnv with the module path and class name for the environment file in your project.

Examples

We included examples using PPO for continous and discrete action environments with slightly adapted CleanRL for recent versions of Gymnasium. Note that CleanRL dependencies should be installed separately before running.

python ./examples/cleanrl_ppo_continous.py

python ./examples/cleanrl_ppo_discrete.py

Use TensorBoard to view the results.

tensorboard --logdir ./runs

Environments

See the src files and original or-gym repository for detailed descriptions of the environments and their operations research background.

• Newsvendor-v0: Multi-period newsvendor problem with stochastic demand, lead times, holding costs, and lost-sales penalties.
• TSP-v0: Sparse, bidirectional traveling-salesperson graph with uniform movement costs and optional action masking.
• TSP-v1: Fully connected traveling-salesperson graph with Euclidean distance costs and penalties for revisiting nodes.
• Knapsack-v0: Unbounded knapsack problem where items can be selected repeatedly until capacity is reached or exceeded.
• Knapsack-v1: Binary knapsack problem where each item can be selected at most once.
• Knapsack-v2: Bounded knapsack problem where each item has a limited quantity available.
• Knapsack-v3: Online knapsack problem where randomly presented items must be accepted or rejected one at a time.
• BinPacking-v0: Small online bin packing instance with bounded waste, stochastic item arrivals, and capacity-based placement actions.
• BinPacking-v1: Large bounded-waste bin packing instance with higher bin capacity, more item sizes, and a longer horizon.
• BinPacking-v2: Small perfectly packable bin packing instance with linear waste rewards.
• BinPacking-v3: Large perfectly packable bin packing instance with linear waste rewards.
• BinPacking-v4: Small perfectly packable bin packing instance with bounded waste rewards.
• BinPacking-v5: Large perfectly packable bin packing instance with bounded waste rewards.
• VMPacking-v0: Online virtual-machine packing problem that assigns CPU and memory demands to physical machines without overloading them.
• VMPacking-v1: Temporary virtual-machine packing problem where assigned processes expire and release physical-machine capacity.
• InvManagement-v0: Multi-period, multi-echelon inventory management system with production capacities, lead times, and backlogged unmet demand.
• InvManagement-v1: Multi-period, multi-echelon inventory management system where unmet demand is treated as lost sales.
• NetworkManagement-v0: Multi-period supply-network inventory management over a directed graph with production, distribution, raw-material, and market nodes.
• NetworkManagement-v1: Supply-network inventory management variant where unmet market demand and replenishment orders are lost instead of backlogged.
• PortfolioOpt-v0: Multi-period portfolio optimization problem for buying and selling three risky assets with transaction costs.
• VehicleRouting-v0: Dynamic food-delivery vehicle routing problem with stochastic orders, pickup and delivery actions, capacity limits, and time penalties.

Reference

@misc{HubbsOR-Gym,
    author={Christian D. Hubbs and Hector D. Perez and Owais Sarwar and Nikolaos V. Sahinidis and Ignacio E. Grossmann and John M. Wassick},
    title={OR-Gym: A Reinforcement Learning Library for Operations Research Problems},
    year={2020},
    Eprint={arXiv:2008.06319}
}

@misc{towers2024gymnasium,
  author={Towers, Mark and Kwiatkowski, Ariel and Terry, Jordan and Balis, John U and De Cola, Gianluca and Deleu, Tristan and Goul{\~a}o, Manuel and Kallinteris, Andreas and Krimmel, Markus and KG, Arjun and others},
  title={Gymnasium: A Standard Interface for Reinforcement Learning Environments},
  year={2024},
  Eprint={arXiv:2407.17032}
}

@article{huang2022cleanrl,
  author  = {Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},
  title   = {CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {274},
  pages   = {1--18},
  url     = {http://jmlr.org/papers/v23/21-1342.html}
}

To cite this repository

@misc{gao2026actorpda,
  author={Gao, Ji and Ju, Caleb and Lan, Guanghui and Tong, Zhaohui},
  title={Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces},
  year={2026},
  Eprint={arXiv:2603.10199}
}