Thread safe and atomic data collection into csv-files
Project description
Simple Data Collector
Thread-safe and atomic collection of tabular data into csv-files.
The simple-data-collector provides a single class with with following methods:
- save
- append
- load
- remove
It was created as a utility function for the Hyperactive-package. It was intended to be used as a search-data collection tool. The search-data can be collected during the optimization run as a dictionary via append
or after the run as a dataframe with the save
-method.
The append
-method is thread-safe to work with hyperactive-multiprocessing. The save
-method is atomic to avoid accidental data-loss.
The simple-data-collector handles functions in the data by converting them to strings. If the data is loaded you can pass the search_space to convert the strings back to functions.
Installation
pip install simple-data-collector
Example
import numpy as np
from hyperactive import Hyperactive
from simple_data_collector import DataCollector
collector = DataCollector("./search_data.csv") # the csv is created automatically
def ackley_function(para):
x, y = para["x"], para["y"]
loss = (
-20 * np.exp(-0.2 * np.sqrt(0.5 * (x * x + y * y)))
- np.exp(0.5 * (np.cos(2 * np.pi * x) + np.cos(2 * np.pi * y)))
+ np.exp(1)
+ 20
)
data_dict = para.para_dict
data_dict["score"] = -loss
collector.append(data_dict) # you can append a dictionary to the csv
return -loss
search_space = {
"x": list(np.arange(-10, 10, 0.01)),
"y": list(np.arange(-10, 10, 0.01)),
}
hyper = Hyperactive()
hyper.add_search(ackley_function, search_space, n_iter=3000)
hyper.run()
search_data = hyper.search_data(ackley_function)
# collector.save(search_data) # save a dataframe instead of appending a dictionary
search_data_l = collector.load(search_space) # load data
print(search_data_l)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for simple_data_collector-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e37fcca0ee058d01d5afee7a3bc17977123297f5c5d6aa184bf218fb9fce618e |
|
MD5 | 207ceaeaf2e7faa90fbd03bf81a18e10 |
|
BLAKE2b-256 | b5e007e099627117c9b9bfe457d3767e0b87a4d6943eee9d8cdc1eaaf92a3585 |