mydatapreprocessing 2.0.4

Library/framework for making predictions.

mydatapreprocessing

Load data from web link or local file (json, csv, excel file, parquet, h5...), consolidate it (resample data, clean NaN values, do string embedding) derive new featurs via columns derivation and do preprocessing like standardization or smoothing. If you want to see how functions works, check it's docstrings - working examples with printed results are also in tests - visual.py.

Links

Repo on github

Official readthedocs documentation

Installation

Python >=3.6 (Python 2 is not supported).

Install just with

pip install mydatapreprocessing

There are some libraries that not every user will be using (for some data inputs). If you want to be sure to have all libraries, you can download requirements_advanced.txt and then install advanced requirements with pip install -r requirements_advanced.txt.

Examples

You can use live jupyter demo on binder

import mydatapreprocessing as mdp

Load data

You can use

• python formats (numpy.ndarray, pd.DataFrame, list, tuple, dict)
• local files
• web urls

You can load more data at once in list.

Syntax is always the same.

data = mdp.load_data.load_data(
    "https://www.ncdc.noaa.gov/cag/global/time-series/globe/land_ocean/ytd/12/1880-2016.json",
    request_datatype_suffix=".json",
    data_orientation="index",
    predicted_table="data",
)
# data2 = mdp.load_data.load_data([PATH_TO_FILE.csv, PATH_TO_FILE2.csv])

Consolidation

If you want to use data for some machine learning models, you will probably want to remove Nan values, convert string columns to numeric if possible, do encoding or keep only numeric data and resample.

data_consolidated = mdp.preprocessing.data_consolidation(
    data, predicted_column=0, remove_nans_threshold=0.9, remove_nans_or_replace="interpolate"
)

Feature engineering

Functions in feature_engineering and preprocessing expects that data are in form (n_samples, n_features). n_samples are ususally much bigger and therefore transformed in data_consolidation if necessary.

Extend original data with

data_extended = mdp.feature_engineering.add_derived_columns(data_consolidated, differences=True, rolling_means=32)

Preprocessing

preprocess_data returns preprocessed data, but also last undifferenced value and scaler for inverse transformation, so unpack it with _

data_preprocessed, _, _ = mdp.preprocessing.preprocess_data(
    data_extended,
    remove_outliers=True,
    smoothit=False,
    correlation_threshold=False,
    data_transform=False,
    standardizeit="standardize",
)

Creating inputs

Create models inputs with

seqs, Y, x_input, test_inputs = mdp.create_model_inputs.make_sequences(
    data_extended.values, predicts=7, repeatit=3, n_steps_in=6, n_steps_out=1, constant=1
)