DaMa-ML 1.0a1

Overview

Dama ML is a framework for data management and is used to do data science and machine learning’s pipelines, also dama-ml try to unify diverse data sources like csv, sql db, hdf5, zarr, etc, and also unify machine learning frameworks (sklearn, Keras, LigthGBM, etc) with a simplify interface.

For more detail read the docs.

Installation

git clone https://github.com/elaeon/dama_ml.git
pip install dama_ml/

or

pip install DaMa-ML

You can install the python dependences with pip, but we strongly recommend install the dependences with conda and conda forge.

conda config --add channels conda-forge
conda create -n new_environment --file dama_ml/requirements.txt
conda activate new_environment
pip install DaMa-ML

Quick start

Configure the data paths where all data will be saved. This can be done with help of dama_ml cli tools.

$ dama-cli config --edit

This will display a nano editor where you can edit data_path, models_path, code_path, class_path and metadata_path.

• data_path is where all datasets are saved.

• models_path is where all files from your models are saved.

• code_path is the repository of code. (In development)

• metadata_path is where the metadata database is saved.

Building a dataset

from dama.data.ds import Data
from dama.drivers.core import Zarr, HDF5
import numpy as np

array_0 = np.random.rand(100, 1)
array_1 = np.random.rand(100,)
array_2 = np.random.rand(100, 3)
array_3 = np.random.rand(100, 6)
array_4 = (np.random.rand(100)*100).astype(int)
array_5 = np.random.rand(100).astype(str)
with Data(name=name, driver=Zarr(mode="w")) as data:
    data.from_data({"x": array_0, "y": array_1, "z": array_2, "a": array_3, "b": array_4, "c": array_5})

We can use a regression model, in this case we use RandomForestRegressor

from dama.reg.extended.w_sklearn import RandomForestRegressor
from dama.utils.model_selection import CV

data.driver.mode = "r"  # we changed mode "w" to "r" to not overwrite the data previously saved
with data, Data(name="test_from_hash", driver=HDF5(mode="w")) as ds:
    cv = CV(group_data="x", group_target="y", train_size=.7, valid_size=.1)  # cross validation class
    stc = cv.apply(data)
    ds.from_data(stc, from_ds_hash=data.hash)
    reg = RandomForestRegressor()
    model_params = dict(n_estimators=25, min_samples_split=2)
    reg.train(ds, num_steps=1, data_train_group="train_x", target_train_group='train_y',
              data_test_group="test_x", target_test_group='test_y', model_params=model_params,
              data_validation_group="validation_x", target_validation_group="validation_y")
    reg.save(name="test_model", model_version="1")

Using RandomForestRegressor to do predictions is like this:

with RandomForestRegressor.load(model_name="test_model", model_version="1") as reg:
    for pred in reg.predict(data):
        prediction = pred.batch.to_ndarray()

CLI

dama-ml has a CLI where you can view your datasets and models. For example

dama-cli datasets

Return a table of datasets previously saved.

Using metadata ..../metadata/metadata.sqlite3
Total 2 / 2

hash                    name            driver    group name    size       num groups  datetime UTC
---------------------  --------------  --------  ------------  --------  ------------  -------------------
sha1.3124d5f16eb0e...  test_from_hash  HDF5      s/n           9.12 KB              6  2019-02-27 19:39:00
sha1.e832f56e33491...  reg0            Zarr      s/n           23.68 KB             6  2019-02-27 19:39:00

dama-cli models

Total 3 / 3
from_ds                       name      group_name    model                                                version     score name        score
-------------------------  ----------  ------------  -------------------------------------------------  ---------  ---------------  ----------
sha1.d8ff5a342d2d7229...  test_model  s/n           dama.reg.extended.w_sklearn.RandomForestRegressor          1  mse               0.162365
sha1.d8ff5a342d2d7229...  test_model  s/n           dama.reg.extended.w_sklearn.RandomForestRegressor          1  msle              0.0741331
sha1.d8ff5a342d2d7229...  test_model  s/n           dama.reg.extended.w_sklearn.RandomForestRegressor          1  gini_normalized  -0.307407

You can use “–help” for view more options.