Library/framework for making predictions.
Project description
mydatapreprocessing
Library contain 3 modules - preprocessing, inputs and generatedata.
Preprocessing
First - preprocessing load data, consolidate it and do the preprocessing. It contains functions like load_data, data_consolidation, preprocess_data, preprocess_data_inverse, add_frequency_columns, rolling_windows, add_derived_columns etc.
Example
import mydatapreprocessing.preprocessing as mdpp
data = "https://blockchain.info/unconfirmed-transactions?format=json"
# Load data from file or URL
data_loaded = mdpp.load_data(data, request_datatype_suffix=".json", predicted_table='txs')
#Some examples of other inputs to data_load function
# myarray_or_dataframe # Numpy array or Pandas.DataFrame
# r"/home/user/my.json" # Local file. The same with .parquet, .h5, .json or .xlsx. On windows it's necessary to use raw string - 'r' in front of string because of escape symbols \
# "https://yoururl/your.csv" # Web url (with suffix). Same with json.
# "https://blockchain.info/unconfirmed-transactions?format=json" # In this case you have to specify also 'request_datatype_suffix': "json", 'data_orientation': "index", 'predicted_table': 'txs',
# {'col_1': [3, 2, 1, 0], 'col_2': ['a', 'b', 'c', 'd']} # Dict with colums or rows (index) - necessary to setup data_orientation!
# You can use more files in list and data will be concatenated. It can be list of paths or list of python objects. Example:
# [{'col_1': 3, 'col_2': 'a'}, {'col_1': 0, 'col_2': 'd'}] # List of records
# [np.random.randn(20, 3), np.random.randn(25, 3)] # Dataframe same way
# ["https://raw.githubusercontent.com/jbrownlee/Datasets/master/daily-min-temperatures.csv", "https://raw.githubusercontent.com/jbrownlee/Datasets/master/daily-min-temperatures.csv"] # List of URLs
# ["path/to/my1.csv", "path/to/my1.csv"]
# Transform various data into defined format - pandas dataframe - convert to numeric if possible, keep
# only numeric data and resample ifg configured. It return array, dataframe
data_consolidated = mdpp.data_consolidation(
data_loaded, predicted_column="weight", data_orientation="index", remove_nans_threshold=0.9, remove_nans_or_replace='interpolate')
# You can add some extra informations to the data that can help (beware it can slow down the machine learning model)
to_be_extended = np.array([[0, 2] * 64, [0, 0, 0, 5] * 32]).T
extended = mdpp.add_frequency_columns(to_be_extended, window=8)
to_be_extended2 = pd.DataFrame([range(30), range(30, 60)]).T
extended2 = mdpp.add_derived_columns(to_be_extended2, differences=True, second_differences=True, multiplications=True,
rolling_means=True, rolling_stds=True, mean_distances=True, window=10)
# Feature extraction is under development :[
# Preprocess data. It return preprocessed data, but also scaler for inverse
# transformation, so unpack it with _
data_preprocessed, _ = mdpp.preprocess_data(data_consolidated, remove_outliers=True, smoothit=False,
correlation_threshold=False, data_transform=False, standardizeit='standardize')
Inputs
Second module is inputs. It take tabular time series data and put it into format (input vector X, output vector y and input for predicted value x_input) that can be inserted into machine learning models for example on sklearn or tensorflow. It contain functions make_sequences, create_inputs and create_tests_outputs
Example
import mydatapreprocessing.inputs as mdpi
data = np.array([[1, 3, 5, 2, 3, 4, 5, 66, 3]]).T
seqs, Y, x_input, test_inputs = mdpi.inputs.make_sequences(data, predicts=7, repeatit=3, n_steps_in=6, n_steps_out=1, constant=1)
Third module is generatedata. It generate some basic data like sin, ramp random. In the future, it will also import some real datasets for models KPI.
generatedata
Third module generate data that can be used for validating machine learning time series prediction results. It can define for example sig, sign, ramp signal or download ECG heart signal.
Example
import mydatapreprocessing as mdp
data = mdp.generatedata.gen_sin(1000)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for mydatapreprocessing-1.1.32.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | ee6348a4a7c8b3c0c1e238b06b4e25e0758080915fdbf109926a4cceafaf8a10 |
|
MD5 | 634ac2a9a7b09831d9dc4e88d05ac6a0 |
|
BLAKE2b-256 | e6ab338883264a7c9c53e7dc9bd6823245d7c130834a6e1829c7a20ac30eb5a9 |
Hashes for mydatapreprocessing-1.1.32-py3.7.egg
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5833d23d5b0bd52318c21b26bd7c99305bc0a24c811b544abfcadab099c6b8dd |
|
MD5 | 5264a456fc1e65523c32f53af98c8053 |
|
BLAKE2b-256 | bf2a3511f960d2149a9f660ef7c0bc8d4d8848e6b734afe510625d31c14ed9fc |
Hashes for mydatapreprocessing-1.1.32-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | abbe29f051b6ae4d77c1bbe76e9b87aa366997bbc26c240bbfc369618f31aae0 |
|
MD5 | 9be86b80efc594cb4c0cffce62900bd6 |
|
BLAKE2b-256 | 8bd5e5d3fc5112ab1299671c6a23993e52e535d49be78a6c155e8951f695d9d2 |