Skip to main content

Thread safe and atomic data collection into csv-files

Project description

Search Data Collector

img not loaded: try F5 :) img not loaded: try F5 :)

Thread-safe and atomic collection of tabular data into csv-files.


The search-data-collector provides a single class with with following methods:

  • save
  • append
  • load
  • remove

It was created as a utility function for the Hyperactive-package. It was intended to be used as a search-data collection tool. The search-data can be collected during the optimization run as a dictionary via append or after the run as a dataframe with the save-method.
The append-method is thread-safe to work with hyperactive-multiprocessing. The save-method is atomic to avoid accidental data-loss.
The search-data-collector handles functions in the data by converting them to strings. If the data is loaded you can pass the search_space to convert the strings back to functions.


Installation

pip install search-data-collector 

Example

import numpy as np
from hyperactive import Hyperactive
from search_data_collector import DataCollector

collector = DataCollector("./search_data.csv") # the csv is created automatically


def ackley_function(para):
    x, y = para["x"], para["y"]

    loss = (
        -20 * np.exp(-0.2 * np.sqrt(0.5 * (x * x + y * y)))
        - np.exp(0.5 * (np.cos(2 * np.pi * x) + np.cos(2 * np.pi * y)))
        + np.exp(1)
        + 20
    )

    data_dict = para.para_dict
    data_dict["score"] = -loss
    collector.append(data_dict)  # you can append a dictionary to the csv

    return -loss


search_space = {
    "x": list(np.arange(-10, 10, 0.01)),
    "y": list(np.arange(-10, 10, 0.01)),
}


hyper = Hyperactive()
hyper.add_search(ackley_function, search_space, n_iter=3000)
hyper.run()
search_data = hyper.search_data(ackley_function)

# collector.save(search_data) # save a dataframe instead of appending a dictionary

search_data_l = collector.load(search_space)  # load data

print(search_data_l)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

search_data_collector-0.2.0-py3-none-any.whl (12.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page