pystopt 6.3

StOpt Python bindings (C++ / pybind11)

The STochastic OPTimization library (StOpt) aims at providing tools in C++ for solving some stochastic optimization problems encountered in finance or in the industry. A python binding is available for some C++ objects provided permitting to easily solve an optimization problem by regression or tree methods. The possibility to create a R package for the regressors is also provided for linux and mac users. Different methods are available :

• dynamic programming methods based on Monte Carlo with regressions (global, local , kernel and sparse regressors), for underlying states following some uncontrolled Stochastic Differential Equations (python binding provided). Transition problems can be solved by:
  • discretization of commands,
  • resolution of some LP and then the Bellman values are approximated by cuts
• dynamic programming with a representation of uncertainties with a tree : transition problems are here solved by
  • some discretizations of the commands,
  • resolution of LP with cut representation of the Bellman values.
• Semi-Lagrangian methods for Hamilton Jacobi Bellman general equations for underlying states following some controlled Stochastic Differential Equations (C++ only)
• Stochastic Dual Dynamic Programming methods to deal with stochastic stocks management problems in high dimension. A SDDP module in python is provided. To use this module, the transitional optimization problem has to written in C++ and mapped to python (examples provided). Uncertainties can be given by Monte Carlo and can be represented by a state with a finite number of values (tree).
• Some branching nesting methods to solve very high dimensional non linear PDEs and some appearing in HJB problems. Besides some methods are provided to solve by Monte Carlo some problems where the underlying stochastic state is controlled. For each method, a framework is provided to optimize the problem and then simulate it out of the sample using the optimal commands previously calculated. Parallelization methods based on OpenMP and MPI are provided in this framework permitting to solve high dimensional problems on clusters. The library should be flexible enough to be used at different levels depending on the user's willingness.

############################################################################################

GENERAL DOCUMENTATION AVAILABLE AT :

https://www.researchgate.net/project/The-STochastic-OPTimization-library-https-gitlabcom-stochastic-control-StOpt

or pick up latest version in :

https://hal.archives-ouvertes.fr/hal-01361291

############################################################################################

Please, while using the library for some research purpose, please cite:

@article{gevret2018stochastic, title={STochastic OPTimization library in C++}, author={Gevret, Hugo and Langren{'e}, Nicolas and Lelong, Jerome and Warin, Xavier and Maheshwari, Aditya}, year={2018} }

###########################################################################################

Licence : The library is published under the GNU LESSER GENERAL PUBLIC LICENSE (see LICENCE.txt, COPYING, COPYING.LESSER files)

###########################################################################################

News (2022/02) : from V5.0, MPI interface changes permitting to have two levels of MPI parallelization in a single software.

###########################################################################################

The library can be built on Linux/Mac/Windows using some recent compilers. It has been tested with

• gcc, clang, intel icc, intel DPC++ on linux,
• clang on mac
• visual studio on windows (VS2019, VS2022)
• intel DPC++ on windows

############################################################################################

############################################################################################

##################

Prerequisite

##################

StOpt library uses some external library. One is included in the package so do no need any installation :

• geners library for generic serialization for C++ : http://geners.hepforge.org/

The following libraries need to be installed :

• zlib, (geners serialization, already present in linux)
• bzip2, (geners serialization, already present in linux)
• eigen library version 3.4 or above : http://http://eigen.tuxfamily.org/
• pybind library https://github.com/pybind/pybind11
• boost library : http://www.boost.org/
• python 3 with numpy package. Scipy package needed for test cases. Mpi4py package needed for test with mpi. Other libraries are optional :
• mpi http://www.open-mpi.org/ for MPI calculation, MS mpi on windows
• COIN OSI, COIN CLP, COIN UTILS only for SDDP test (http://www.coin-or.org/)

StOpt relies on cmake for compilation http://www.cmake.org/ and permits to generate projects (Kdevelopp or Visual Studio for example).

##################################

Important for multithreading

##################################

StOpt uses multithread : so OMP_NUM_THREADS environment variable has to be set to a number of threads to use.

##################################

Important to use python

##################################

Python test case are located in StOpt/test/python. To use python the PYTHONPATH and LD_LIBRARY_PATH should be updated to include the directory containing the StOpt (.so, .pyd) library (default is the lib directory where the library is built). If the build is achieved in /home/ME/GIT_REPOS/BUILD_RELEASE_GCC then under linux and MacOS:

LD_LIBRARY_PATH=/home/ME/GIT_REPOS/BUILD_RELEASE_GCC/lib/:$LD_LIBRARY_PATH
PYTHONPATH=/home/ME/GIT_REPOS/BUILD_RELEASE_GCC/lib:$PYTHONPATH

####################################

INSTALLATION, TESTS LINUX/MAC

#################################### ####################################

Install package for zlib, bzip2, python, scipy (http://www.scipy.org/), eigen, boost (including boost mpi if necessary), cmake and cmake gui (ccmake) using your favorite package manager (apt, snap, yaourt, .. on linux, homebrew on Mac OS) Go to StOpt directory (containing directories geners-1.10.0, StOpt ...)

cd ..  
mkdir BUILD  
cd BUILD
cmake ../StOpt
cmake ../StOpt

(Rerun cmake so that the python libs can be found..) If packages for libraries are not in usual place, use ccmake (ccmake ../StOpt) to define included directories and libraries needed. In cmake gui define :

• if python bindings should be compiled (default no)
• if the library should use mpi (default true)
• if the library should compile sddp test (default false (COIN CLP needed)) This gui can be called by

ccmake ../StOpt
make

Test cases are run by:

ctest

A package can be build using cpack :

• for debian : cpack -G DEB
• for Red Hat : cpack -G RPM
• a GZIP compression : cpack -G TGZ

Installation in /usr/local

make install

For installation in given directory /home/toto/test (installation will be achieved in /home/toto/test/usr/local)

make DESTDIR=/home/toto/test install

It is also possible to have installation in /home/toto/test such that the directory tree looks like /home/toto/test/lib, /home/toto/test/include...

cmake -DCMAKE_INSTALL_PREFIX=/home/toto/test ../StOpt
make install

##########################################################################

Python test

##########################################################################

To run the python tests if library compiled with python (don't forget to update PYTHONPATH in your .bashrc):

cd ../StOpt/test/python
python -m unittest discover

In StOpt/test/python/functional python version of C++ test cases are available.

Two kind of mappings are available.

• A first mapping is achieved at the grids and conditional expectation level.
• A second one is achieved at a time step resolution level (file HighLevel.py)

In order to test python mpi :

mpirun -np x python test*HighLevelMpi.py

with "x" number of mpi processus generated.

special case for Mac User

The boost installation can be achieved with a package manager : homebrew for example will install the missing dependencies (openmpi ...) In case of a boost compîlation from source, check that "boost mpi" is compiled and installed. If not : In the directory where the boost file is decompressed,

./boostrap.sh

Edit a file user-config.jam and add (don't forget the space between mpi and ;)

using mpi ;

./bjam --user-config=user-config.jam install

Most of package can be installed with homebrew (eigen, pybind11). Only when activating the SDDP option, Coin utils, osi and clp have to be compiled and installed as the homebrew project only provide binaries.

##################################

INSTALLATION, TESTS WINDOWS :##

################################## ##################################

Minimal Requirement

• Win 10
• Visual Studio 2019

WINDOWS : Download CMake for Windows, git for windows, vcpkg and install them

Compile your own StOpt libs ( when using python bindings) to use the python library

#####################################################################################

Add python dependance in file vcpkg.json in StOpt directory if a special python version is needed

in vcpkg directory,

git rev-parse HEAD

get back builtin-base string

replace the builtin-base in the vcpkg.json file and modify it according to your python version

StOpt compilation and local vcpkg package installation

In the directory where StOpt is cloned :

mkdir BUILD
cd BUILD
SET VCPKG_FEATURE_FLAGS=versions
cmake -DCMAKE_TOOLCHAIN_FILE=VcpkgDir/vcpkg/scripts/buildsystems/vcpkg.cmake -DBUILD_SDDP=ON -DBUILD_DPCUTS=ON  ../StOpt
cmake --build . --config=Release

In order to use DPC++ on windows, ninja (https://ninja-build.org/) should be installed and use the tool chain:

cmake -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icx -GNinja -DCMAKE_TOOLCHAIN_FILE=VcpkgDir/vcpkg/scripts/buildsystems/vcpkg.cmake -DBUILD_SDDP=ON -DBUILD_DPCUTS=ON  ../StOpt
cmake --build . --config=Release

alternative : Create your own StOpt c++ vcpkg package (latest python)

#########################################################################

In vcpkg directory VcpkgDir (absolute path):

vcpkg install eigen3 --triplet x64-windows-release
vcpkg install mpi --triplet x64-windows-release
vcpkg install coinutils  --triplet x64-windows-release
vcpkg install coin-or-osi --triplet x64-windows-release
vcpkg install coin-or-clp --triplet x64-windows-release
vcpkg install boost --triplet x64-windows-release
vcpkg install boost-mpi --triplet x64-windows-release
vcpkg install pthreads --triplet x64-windows-release
vcpkg install python3 --triplet x64-windows-release

To install StOpt library from vcpkg registry, clone the git repo in a directory :

https://gitlab.com/stochastic-control/stopt-vcpkg-registry-with-python

In vcpkg directory :

vcpkg install stopt --overlay-ports=PathToVCPKGRegistry\stopt-vcpkg-registry-with-python\ports\stopt  --triplet x64-windows-release

###############################################################################

Helper for cmake compilation for projects using StOpt

###############################################################################

in utils directory, a file FindStOpt.cmake and a file FindGeners.cmake are provided to help compilation in projects using cmake

################################################################################

R package

###############################################################################

A readme.txt is available in directory StOpt/R

###############################################################################

Docker

###############################################################################

Thanks to Benjamin Texier a Dockerfile is available based on ubuntu and anaconda to test the python interface in a jupyter notebook