A testbed for simulationoptimization experiments.
Project description
Welcome to SimOpt!
SimOpt is a testbed of simulationoptimization problems and solvers. Its purpose is to encourage the development and constructive comparison of simulationoptimization (SO) solvers (algorithms). We are particularly interested in the finitetime performance of solvers, rather than the asymptotic results that one often finds in related literature.
The mostuptodate publication about this library is Eckman et al. (2021).
For the purposes of this project, we define simulation as a very general technique for estimating statistical measures of complex systems. A system is modeled as if the probability distributions of the underlying random variables were known. Realizations of these random variables are then drawn randomly from these distributions. Each replication gives one observation of the system response, i.e., an evaluation of the objective function or stochastic constraints. By simulating a system in this fashion for multiple replications and aggregating the responses, one can compute statistics and use them for evaluation and design.
Several papers have discussed the development of SimOpt and experiments run on the testbed:
 Pasupathy and Henderson (2006) explains the original motivation for the testbed.
 Pasupathy and Henderson (2011) describes an earlier interface for MATLAB implementations of problems and solvers.
 Dong et al. (2017) conducts an experimental comparison of several solvers in SimOpt and analyzes their relative performance.
 Eckman et al. (2019) describes in detail changes to the architecture of the MATLAB version of SimOpt and the control of random number streams.
 Eckman et al. (2021) introduces the design of experiments for comparing solvers; this design has been implemented in the latest Python version of SimOpt. For detailed description of the terminology used in the library, e.g., factors, macroreplications, postprocessing, solvability plots, etc., see this paper.
Code
 The
master
branch contains the source code for the latest version of the testbed, which is written in Python.  The
matlab
branch contains a previous stable version of the testbed written in MATLAB.
Documentation
Full documentation for the source code can be found here.
Getting Started
The most straightforward way to interact with the library is to clone this repository. (If you anticipate making improvements or contributions to SimOpt, you should first fork the repository so that you can later request your changes be integrated via GitHub's pull request feature.) Alternatively, you can choose to install the library as a Python package; see the sections titled Package and Basic Example below for more details about this option.
Download a copy of the cloned repository to your local machine and navigate to the simopt
folder in your preferred integrated development environment (IDE). You will need to make sure that you have the following dependencies installed: Python 3, numpy
, scipy
, matplotlib
, pandas
, seaborn
, and mrg32k3a
. Run the command python m pip install numpy scipy matplotlib pandas seaborn mrg32k3a
to install them from the terminal.
The demo
folder contains a handful of useful scripts that can be easily modified, as directed in the comments.

demo_model.py
: Run multiple replications of a simulation model and report its responses. 
demo_problem.py
: Run multiple replications of a given solution for an SO problem and report its objective function values and lefthand sides of stochastic constraints. 
demo_problem_solver.py
: Run multiple macroreplications of a solver on a problem, save the outputs to a .pickle file in theexperiments/outputs
folder, and save plots of the results to .png files in theexperiments/plots
folder. 
demo_problems_solvers.py
: Run multiple macroreplications of groups of problemsolver pairs and save the outputs and plots. 
demo_data_farming_model.py
: Create a design over model factors, run multiple replications at each design point, and save the results to a comma separated value (.csv) file in thedata_farming_experiments
folder. 
demo_sansscontironorecont_experiment
: Run multiple solvers on multiple versions of (s, S) inventory, iron ore, and stochastic activiy network problems and produce plots.
Graphical User Interface (GUI)  User Guide
Installation
To start up the GUI, navigate to the outer simopt
directory and run the command python3 m simopt.GUI
from the terminal. The GUI depends on Python 3, numpy
, scipy
, matplotlib
, Pillow
, and tkinter
. Run the command python3 m pip install numpy scipy matplotlib Pillow tkinter
to install them from the terminal.
Overview
From the GUI, you can create a specified problemsolver pair or a problemsolver group, run macroreplications, and generate plots. The main page provides ways to create or continue working with experiments:
 Create an individual problemsolver pair with customized problem and solver factors.
 Load a .pickle file of a previously created problemsolver pair.
 Create a problemsolver group.
At the bottom of the main page, there is a workspace containing all problemsolver pairs and problemsolver groups. The first tab lists the problemsolver pairs ready to be run or postreplicated, the second tab lists the problemsolver groups made from the crossdesign or by generating a problemsolver group from partial set of problemsolver pair in the first tab, and the third tab lists those problemsolver pairs that are ready to be postnormalized and prepared for plotting.
1. Creating a problemsolver pair
This is the main way to add problemsolver pairs to the queue in the workspace.
 First, select a solver from the "Solver" dropdown list. Each of the solvers has an abbreviation for the type of problems the solver can handle. Once a solver is selected, the "Problem" list will be sorted and show only the problems that work with the selected solver.
 Change factors associated with the solver as necessary. The first factor is a customizable name for the solver that use can specify.
 All solvers with unique combinations of factors must have unique names, i.e., no two solvers can have the same name, but different factors. If you want to use the same solver twice for a problem but with different solver factors, make sure you change the name of the solver accordingly. For example, if you want to create two problemsolver pairs with the same problem and solver but with or without CRN for the solver, you can change the name of the solver of choice for each pair to reflect that. This name will appear in the queue within the workspace below.
 Select a problem from the "Problem" dropdown list.
Each problem has an abbreviation indicating which types of solver is compatible to solve it. The letters in the abbreviation stand for:
Objective Constraint Variable Direct Gradient Observations Single (S) Unconstrained (U) Discrete (D) Available (G) Multiple (M) Box (B) Continuous (C) Not Available (N) Deterministic (D) Mixed (M) Stochastic (S)  Change factors associated with the problem and model as necessary.
 All problems with unique factors must have unique names, i.e., no two problems can have the same name, but different factors. If you want to use the same problem twice for a solver but with different problem or model factors, make sure you change the name of the problem accordingly. This name will appear in the queue within the workspace below.
 The number of macroreplications can be modified in the topleft corner. The default is 10.
 Select the "Add problemsolver pair" button, which only appears when a solver and problem is selected. The problemsolver pair will be added in the "Queue of problemsolver pairs."
2. Loading a problemsolver pair from a file
Instead of creating a problemsolver pair from scratch, you can load one from a *.pickle file:
 In the top left corner, click "Load a problemsolver pair". Your file system will pop up, and you can navigate to and select an appropriate *.pickle file. The GUI will throw an error if the selected file is not a *.pickle file.
 Once a problemsolver pair object is loaded, it will be added to the "Queue of problemsolver pairs".
 The Run and PostProcess buttons will be updated to accurately reflect whether the problemsolver pair has already been run and/or postprocessed.
3. Creating a problemsolver group
Currently, problemsolver groups can only be created within the GUI or command line; they cannot be loaded from a file.
You can create a problemsolver group and add a new item to the "Queue of problemsolver groups" in two ways. The first is a quick grouping of problems and solvers that are compatible with their default factors: of problems and solvers with their default factors.
 Click the "Create a problemsolver group" button.
 Check the compatibility of the Problems and Solvers being selected. Note that solvers with deterministic constraint type can not handle problems with stochastic constraints (e.g., ASTRODF cannot be run on FACSIZE2).
 Specify the number of macroreplications  the default is 10.
 Click "Confirm CrossDesign problemsolver group."
 The popup window will disappear, and the problemsolver pairs frame will automatically switch to the "Queue of problemsolver groups".
 To exit out of the problemsolver group popup without creating a problemsolver group, click the red "x" in the topleft corner of the window.
The second is converting a list of problemsolver pairs into a problemsolver group by a crossdesign:
 Select the problemsolver pairs of interest from the "Queue of problemsolver pairs".
 Clicking the "Convert to a problemsolver group" button. This will complete the crossdesign for the partial list and create a new row in the "Queue of problemsolver groups".
Running a problemsolver pair or a problemsolver group
To run a problemsolver pair or a problemsolver group, click the "Run" button in the "Queue of problemsolver pairs" or "Queue of problemsolver groups". Once the problemsolver pair or problemsolver group has been run, the "Run" button becomes disabled. Note: Running a problemsolver pair can take anywhere from a couple of seconds to a couple of minutes depending on the problemsolver pair and the number of macroreplications.
PostProcessing and PostNormalization
Postprocessing happens before postnormalizing and after the run is complete. You can specify the number of postreplications and the (proxy) optimal solution or function value. After postnormalization is complete, the Plotting window appears. To exit out of the PostProcess/Normalize popup without postprocessing or postnormalizing, click the red "x" in the topleft corner of the window.
 problemsolver pair
problemsolver pairs can be postprocessed from the "Queue of problemsolver pairs" tab by clicking "PostProcess." Adjust PostProcessing factors as necessary. Only problemsolver pairs that have already been run can be postprocessed. After postprocessing, click the "PostNormalize by Problem" tab to select which problemsolver pairs to postnormalize together.
 Only problemsolver pairs with the same problem can be postnormalized together.
 Once all problemsolver pairs of interest are selected, click the "PostNormalize Selected" button at the bottom of the GUI (this button only appears when in the PostNormalize tab).
 In the new popup form, update any values necessary and click "PostNormalize" when the problemsolver pairs are ready to be postnormalized.
 problemsolver group
problemsolver groups are postprocessed and postnormalized at the same time. In the "Queue of problemsolver groups" tab, click the "PostProcess" button for the specific problemsolver group, then change any values necessary, then click "PostProcess".
Plotting
The Plotting page is identical for both problemsolver pairs and problemsolver groups. Currently, multiple problemsolver pairs with the same problem can be plotted together, and any problemsolver pair from a single problemsolver group can be plotted together:
 On the left side, select one or more problems from the problem list.
 Select solvers from the solver list.
 On the right side, select a plot type and adjust plot parameters and settings. There are 5 settings common to most plot types: Confidence Intervals, Number of Bootstrap Samples, Confidence Level, Plot Together, and Print Max HW. The type of plots that are currently available in the GUI are: Mean Progress Curve, Quantile Progress Curve, Solve Time CDF, Scatter Plot, CDF Solvability, Quantile Solvability, CDF Difference Plot, Quantile Difference Plot, Terminal Box/Violin, and Terminal Scatter.
 Click "Add."
 All plots will show in the plotting queue, along with information about their parameters and where the file is saved.
 To view one plot, click "View Plot." All plots can be viewed together by clicking "See All Plots" at the bottom of the page.
 To return to the main page, click the red "x" in the topleft corner of the window.
Contributing
You can contribute problems and solver to SimOpt by using pull requests in GitHub or corresponding with the developers.
Package
The simoptlib
package is available to download through the Python Packaging Index (PyPI) and can be installed from the terminal with the following command:
python m pip install simoptlib
Basic Example
After installing simoptlib
, the package's main modules can be imported from the Python console (or in code):
import simopt
from simopt import models, solvers, experiment_base
The following snippet of code will run 10 macroreplications of the Random Search solver ("RNDSRCH") on the Continuous Newsvendor problem ("CNTNEWS1"):
myexperiment = simopt.experiment_base.ProblemSolver("RNDSRCH", "CNTNEWS1")
myexperiment.run(n_macroreps=10)
The results will be saved to a .pickle file in a folder called experiments/outputs
. To postprocess the results, by taking, for example 200 postreplications at each recommended solution, run the following:
myexperiment.post_replicate(n_postreps=200)
simopt.experiment_base.post_normalize([myexperiment], n_postreps_init_opt=200)
A .txt file summarizing the progress of the solver on each macroreplication can be produced:
myexperiment.log_experiment_results()
A .txt file called RNDSRCH_on_CNTNEWS1_experiment_results.txt
will be saved in a folder called experiments/logs
.
One can then plot the mean progress curve of the solver (with confidence intervals) with the objective function values shown on the yaxis:
simopt.experiment_base.plot_progress_curves(experiments=[myexperiment], plot_type="mean", normalize=False)
The Python scripts in the demo
folder provide more guidance on how to run common experiments using the library.
One can also use the SimOpt graphical user interface by running the following from the terminal:
python m simopt.GUI
Authors
The core development team currently consists of
 David Eckman (Texas A&M University)
 Sara Shashaani (North Carolina State University)
 Shane Henderson (Cornell University)
Citation
To cite this work, please use
@misc{simoptgithub,
author = {D. J. Eckman and S. G. Henderson and S. Shashaani and R. Pasupathy},
title = {{SimOpt}},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/simoptadmin/simopt}},
commit = {eda24b9f6a5885a37321ad7f8534bf10dec22480}
}
Acknowledgments
An earlier website for SimOpt, http://www.simopt.org, was developed through work supported by the National Science Foundation under grant nos. DMI0400287, CMMI0800688 and CMMI1200315. Recent work on the development of SimOpt has been supported by the National Science Foundation under grant nos. DGE1650441, CMMI1537394, CMMI1254298, CMMI1536895, CMMI2206972, IIS1247696, and TRIPODS+X DMS1839346, by the Air Force Office of Scientific Research under grant nos. FA95501210200, FA95501510038, and FA95501610046, and by the Army Research Office under grant no. W911NF1710094. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation (NSF).
Project details
Release history Release notifications  RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for simoptlib1.0.2py3noneany.whl
Algorithm  Hash digest  

SHA256  7f1fb197f33a3b89495321cbe0ccc44fcf8ac4b976f6aa78f8621bbe6d25cf72 

MD5  ead8afe8b82fee801110d513eeb04fdd 

BLAKE2b256  2d67145f1e86e305a8f12e7ae47c467b9027f1120b415c3ca961b24f626c004b 