speval 0.1.1

Simple Parellel Function Evaluator

speval

Simple Parallel Function Evaluator for python.

speval is a python package that allows one to easily parallelize multiple evaluations of some time consuming function. speval is designed to have no extra dependencies and be extra simple to use.

Usage

Let say we have a long running function eval_me and we would like to evaluate it 100 times, each time with different parameters. And, moreover perform these evaluations in parallel. We can easily achieve this with a simple python script of the form:

from speval import speval

list_of_args = [
   args_for_eval1,
   args_for_eval2,
   ...
   args_for_eval100,
]

speval(eval_me, list_of_args, db_name)

Now, we can run one instance of this script and it will start evaluating function eval_me for each point of list_of_args. To parallelize evaluations we just need to run another N instances of this script, and they are going to distribute evaluations of eval_me among them, using sqlite3 database db_name for synchronization.

Once all evaluations are complete we can use function load_evals to retrieve results(if any) of each evaluation from the database db_name.

Motivation

speval was written because the author needed a simple hyperparameter optimization package without heavy dependencies(like mongodb), or need to properly set up workers on multiple machines.

Requirements

python-3 built with sqlite3 support(enabled by default)

Examples

A number of examples is stored in the examples subdirectory. Here is a brief overview of them.

01. Hello World

Let’s start with a primitive function to evaluate

import time

def func(arg):
    print("Hello world: %d" % (arg))
    time.sleep(2)

Say we want to evaluate function func 100 times using evaluation index as an argument to func. To perform this we first need to define a list with arguments for each evaluation

eval_space = list(range(100))

And then call speval

from speval import speval

speval(func, eval_space, "/tmp/01_hello_world.db", timeout = 60)

Here we see that speval takes function to evaluate func as a first argument, list with arguments for each evaluation eval_space as a second argument. We also pass there a sqlite database file name ("/tmp/01_hello_world.db") which will be used for coordination between different processes. And a timeout parameter which tells speval that a given evaluation of function func has failed if it was not completed in 60 seconds and therefore needs to be restarted.

Done. Now we can run the resulting script(e.g. examples/01_hello_world.py) as many times in parallel as many jobs we want to allocate for the task of printing “Hello world”.

For example, on a unix system one may run in a terminal

$ python examples/01_hello_world.py
Hello world: 0
Hello world: 1
Hello world: 2
Hello world: 3
Hello world: 4
...

So, it will print "Hello world: N" each 2 seconds, and without further intervention will finish in about 200 seconds. To parallelize printing we can run another say 20 jobs in a separate terminal. E.g.

$ for i in {1..20}; do (python examples/01_hello_world.py &) ; done

Now, all pending hello worlds will be printed in just about 10 seconds.

02. Storing/Retrieving Function Results

In examples directory there are 2 scripts 02_store_results.py and 02_retrieve_results.py. First script is just a modification of the hello world example, where function func also returns a value, which is automatically stored in the database. Second script there is used to demonstrate how to retrieve saved results from the database.

03. Using speval for Hyperparameter Optimization

Finally, there are another 2 examples 03_fit_line.py and 03_get_best_fit.py, which demonstrate how to use speval for hyperparameter optimization over a predefined search space(grid search or randomized search). Here the first script evaluates objective function on each point of the search space. This script can be run on the multiple machines in parallel, provided they all have access to a shared mount on which a sqlite3 database resides.

The second script is can be used to find optimal values of hyperparameters from the results of these evaluations stored in the database.

Limitations

speval uses sqlite3 internal advisory file locking mechanism to prevent race conditions. This mechanism is known not to work properly for some network filesystem setups. So be advised if your database is on NFS mount.

Additionally, speval relies on json format to serialize parameters. So, you need to make sure that eval_space and results returned by the eval_func are json serializable objects.