partialwrap 2.0

About partialwrap

partialwrap is a Python library providing easy wrapper functions to use with Python’s functools.partial. Partial’s ingenious mechanism allows to use even very complex functions with many arguments and keyword arguments with routines that need functions in the simple form func(x). This includes Python’s map function, the minimizers in scipy.optimize, and plenty of third-party modules such as emcee or pyeee. partialwrap allows to use any external executable as well as any Python function with arbitrary arguments and keywords to be used with functools.partial. It also allows to use the executables or functions with easy parallelization of code with, for example, the parallel map function of the multiprocessing module or via the Message Passing Interface (MPI).

Documentation

The complete documentation for partialwrap is available at Github pages:

https://mcuntz.github.io/partialwrap/

Quick usage guide

partialwrap provides two wrapper functions to work with external executables: partialwrap.exe_wrapper and partialwrap.exe_mask_wrapper.

partialwrap writes the input arguments to the wrapper functions into files that can be read by the external executable. The executable should write its result to a file that will then be read by partialwrap in return. This means partialwrap needs to have a function parameterwriter that writes the parameter file parameterfile needed by the executable exe. It then also needs to have a function outputreader for reading the output file outputfile of the external executable, perhaps calculating an objective function value.

Take the Rastrigin function, which is a popular function for performance testing of optimization algorithms: y = a*n + sum_i^n(x_i^2 - a*cos(b*x_i)). It has a global minimum of 0 at all x_i = 0. a influences mainly the depth of the (local and global) minima, whereas b influences mainly the size of the minima. A common form uses a = 10 and b = 2*pi. The parameters x_i should then be in the interval [-5.12, 5.12].

Consider for simplicity an external Python program (e.g. rastrigin1.py) that calculates the Rastrigin function with a = 10 and b = 2*pi, reading in an arbitrary number of parameters x_i from a parameterfile = params.txt and writing its output into outputfile = out.txt:

# File: rastrigin1.py
import numpy as np
from partialwrap import standard_parameter_reader

# Rastrigin function a=10, b=2*pi
def rastrigin1(x):
    return 10. * len(x) + np.sum(x**2 - 10. * np.cos(2. * np.pi * x))

# read parameters
x = standard_parameter_reader('params.txt')

# calc function
y = rastrigin1(x)

# write output file
with open('out.txt', 'w') as ff:
    print(y, file=ff)

This program can be called on the command line (if params.txt is present) with:

python rastrigin1.py

The external program calculating the Rastrigin function could, of course, also be written in any compiled language such as C or Fortran. See the userguide for details. The external program, here the Python version, can be used with Python’s functools.partial and the wrapper function partialwrap.exe_wrapper:

import scipy.optimize as opt
from functools import partial
from partialwrap import exe_wrapper
from partialwrap import standard_parameter_writer, standard_output_reader

rastrigin_exe   = ['python3', 'rastrigin1.py']
parameterfile   = 'params.txt'
parameterwriter = standard_parameter_writer
outputfile      = 'out.txt'
outputreader    = standard_output_reader
rastrigin_wrap  = partial(exe_wrapper, rastrigin_exe,
                          parameterfile, parameterwriter,
                          outputfile, outputreader, {})

x0  = [0.1, 0.2, 0.3]
res = opt.minimize(rastrigin_wrap, x0, method='BFGS')

The scipy.optimize function minimize() passes its sampled parameters to exe_wrapper, which writes it to the file parameterfile = 'params.txt'. It then calls rastrigin_exe = 'python3 rastrigin1.py' and reads its outputfile = 'out.txt'. partialwrap.standard_parameter_reader and partialwrap.standard_parameter_writer are convenience functions that read and write one parameter per line in a file without a header. The empty dictionary at the end is explained in the userguide.

More elaborate input/output of the external program can simply be done by replacing standard_parameter_reader and standard_parameter_writer with appropriate functions, while the rest stays pretty much the same.

Installation

The easiest way to install is via pip:

pip install partialwrap

or via conda:

conda install -c conda-forge partialwrap

Requirements:

• NumPy

License

partialwrap is distributed under the MIT License. See the LICENSE file for details.

Copyright (c) 2016-2023 Matthias Cuntz

The project structure is based on a template provided by Sebastian Müller.