Skip to main content

Thread safe and atomic data collection into csv-files

Project description

Simple Data Collector

img not loaded: try F5 :) img not loaded: try F5 :)

Thread-safe and atomic collection of tabular data into csv-files.


The simple-data-collector provides a single class with with following methods:

  • save
  • append
  • load
  • remove

It was created as a utility function for the Hyperactive-package. It was intended to be used as a search-data collection tool. The search-data can be collected during the optimization run as a dictionary via append or after the run as a dataframe with the save-method.
The append-method is thread-safe to work with hyperactive-multiprocessing. The save-method is atomic to avoid accidental data-loss.
The simple-data-collector handles functions in the data by converting them to strings. If the data is loaded you can pass the search_space to convert the strings back to functions.


Installation

pip install simple-data-collector 

Example

import numpy as np
from hyperactive import Hyperactive
from simple_data_collector import DataCollector

collector = DataCollector("./search_data.csv") # the csv is created automatically


def ackley_function(para):
    x, y = para["x"], para["y"]

    loss = (
        -20 * np.exp(-0.2 * np.sqrt(0.5 * (x * x + y * y)))
        - np.exp(0.5 * (np.cos(2 * np.pi * x) + np.cos(2 * np.pi * y)))
        + np.exp(1)
        + 20
    )

    data_dict = para.para_dict
    data_dict["score"] = -loss
    collector.append(data_dict)  # you can append a dictionary to the csv

    return -loss


search_space = {
    "x": list(np.arange(-10, 10, 0.01)),
    "y": list(np.arange(-10, 10, 0.01)),
}


hyper = Hyperactive()
hyper.add_search(ackley_function, search_space, n_iter=3000)
hyper.run()
search_data = hyper.search_data(ackley_function)

# collector.save(search_data) # save a dataframe instead of appending a dictionary

search_data_l = collector.load(search_space)  # load data

print(search_data_l)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

simple_data_collector-0.2.0-py3-none-any.whl (12.6 kB view details)

Uploaded Python 3

File details

Details for the file simple_data_collector-0.2.0-py3-none-any.whl.

File metadata

  • Download URL: simple_data_collector-0.2.0-py3-none-any.whl
  • Upload date:
  • Size: 12.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.26.0 setuptools/59.4.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.5

File hashes

Hashes for simple_data_collector-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 e37fcca0ee058d01d5afee7a3bc17977123297f5c5d6aa184bf218fb9fce618e
MD5 207ceaeaf2e7faa90fbd03bf81a18e10
BLAKE2b-256 b5e007e099627117c9b9bfe457d3767e0b87a4d6943eee9d8cdc1eaaf92a3585

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page