Simple Data Collector

Thread-safe and atomic collection of tabular data into csv-files.

The simple-data-collector provides a single class with with following methods:

• save
• append
• load
• remove

It was created as a utility function for the Hyperactive-package. It was intended to be used as a search-data collection tool. The search-data can be collected during the optimization run as a dictionary via append or after the run as a dataframe with the save-method.
The append-method is thread-safe to work with hyperactive-multiprocessing. The save-method is atomic to avoid accidental data-loss.
The simple-data-collector handles functions in the data by converting them to strings. If the data is loaded you can pass the search_space to convert the strings back to functions.

Installation

pip install simple-data-collector 

Example

import numpy as np
from hyperactive import Hyperactive
from simple_data_collector import DataCollector

collector = DataCollector("./search_data.csv") # the csv is created automatically


def ackley_function(para):
    x, y = para["x"], para["y"]

    loss = (
        -20 * np.exp(-0.2 * np.sqrt(0.5 * (x * x + y * y)))
        - np.exp(0.5 * (np.cos(2 * np.pi * x) + np.cos(2 * np.pi * y)))
        + np.exp(1)
        + 20
    )

    data_dict = para.para_dict
    data_dict["score"] = -loss
    collector.append(data_dict)  # you can append a dictionary to the csv

    return -loss


search_space = {
    "x": list(np.arange(-10, 10, 0.01)),
    "y": list(np.arange(-10, 10, 0.01)),
}


hyper = Hyperactive()
hyper.add_search(ackley_function, search_space, n_iter=3000)
hyper.run()
search_data = hyper.search_data(ackley_function)

# collector.save(search_data) # save a dataframe instead of appending a dictionary

search_data_l = collector.load(search_space)  # load data

print(search_data_l)