Thread safe and atomic data collection into csv-files
Project description
Simple Data Collector
Thread-safe and atomic collection of tabular data into csv-files.
The simple-data-collector provides a single class with with following methods:
- save
- append
- load
- remove
It was created as a utility function for the Hyperactive-package. It was intended to be used as a search-data collection tool. The search-data can be collected during the optimization run as a dictionary via append
or after the run as a dataframe with the save
-method.
The append
-method is thread-safe to work with hyperactive-multiprocessing. The save
-method is atomic to avoid accidental data-loss.
The simple-data-collector handles functions in the data by converting them to strings. If the data is loaded you can pass the search_space to convert the strings back to functions.
Installation
pip install simple-data-collector
Example
import numpy as np
from hyperactive import Hyperactive
from simple_data_collector import DataCollector
collector = DataCollector("./search_data.csv") # the csv is created automatically
def ackley_function(para):
x, y = para["x"], para["y"]
loss = (
-20 * np.exp(-0.2 * np.sqrt(0.5 * (x * x + y * y)))
- np.exp(0.5 * (np.cos(2 * np.pi * x) + np.cos(2 * np.pi * y)))
+ np.exp(1)
+ 20
)
data_dict = para.para_dict
data_dict["score"] = -loss
collector.append(data_dict) # you can append a dictionary to the csv
return -loss
search_space = {
"x": list(np.arange(-10, 10, 0.01)),
"y": list(np.arange(-10, 10, 0.01)),
}
hyper = Hyperactive()
hyper.add_search(ackley_function, search_space, n_iter=3000)
hyper.run()
search_data = hyper.search_data(ackley_function)
# collector.save(search_data) # save a dataframe instead of appending a dictionary
search_data_l = collector.load(search_space) # load data
print(search_data_l)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
File details
Details for the file simple_data_collector-0.2.0-py3-none-any.whl
.
File metadata
- Download URL: simple_data_collector-0.2.0-py3-none-any.whl
- Upload date:
- Size: 12.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.26.0 setuptools/59.4.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e37fcca0ee058d01d5afee7a3bc17977123297f5c5d6aa184bf218fb9fce618e |
|
MD5 | 207ceaeaf2e7faa90fbd03bf81a18e10 |
|
BLAKE2b-256 | b5e007e099627117c9b9bfe457d3767e0b87a4d6943eee9d8cdc1eaaf92a3585 |