Library/framework for making predictions.
Project description
mydatapreprocessing
Load data web link or local file(json, csv, excel file, parquet, parquet...), consolidate it, do preprocessing like resampling standardization, string embedding, generating new columns, feature extraction etc. based on configuration.
Library contain 3 modules.
First - preprocessing
load data and preprocess it. it contains functions like load_data, data_consolidation, preprocess_data, preprocess_data_inverse, add_frequency_columns, rolling_windows, add_derived_columns
Example
data = "https://blockchain.info/unconfirmed-transactions?format=json"
# Load data from file or URL
data_loaded = mdp.load_data(data, request_datatype_suffix=".json", predicted_table='txs')
# Transform various data into defined format - pandas dataframe - convert to numeric if possible, keep
# only numeric data and resample ifg configured. It return array, dataframe
data_consolidated = mdp.data_consolidation(
data_loaded, predicted_column="weight", data_orientation="index", remove_nans_threshold=0.9, remove_nans_or_replace='interpolate')
# Preprocess data. It return preprocessed data, but also last undifferenced value and scaler for inverse
# transformation, so unpack it with _
data_preprocessed, _, _ = mdp.preprocess_data(data_consolidated, remove_outliers=True, smoothit=False,
correlation_threshold=False, data_transform=False, standardizeit='standardize')
Allowed data formats for load_data are examples
myarray_or_dataframe # Numpy array or Pandas.DataFrame
r"/home/user/my.json" # Local file. The same with .parquet, .h5, .json or .xlsx. On windows it's necessary to use raw string - 'r' in front of string because of escape symbols \
"https://yoururl/your.csv" # Web url (with suffix). Same with json.
"https://blockchain.info/unconfirmed-transactions?format=json" # In this case you have to specify also 'request_datatype_suffix': "json", 'data_orientation': "index", 'predicted_table': 'txs',
[{'col_1': 3, 'col_2': 'a'}, {'col_1': 0, 'col_2': 'd'}] # List of records
{'col_1': [3, 2, 1, 0], 'col_2': ['a', 'b', 'c', 'd']} # Dict with colums or rows (index) - necessary to setup data_orientation!
Second module is inputs. It take tabular time series data and put it into format that can be inserted into machine learning models for example on sklearn or tensorflow. It contain functions make_sequences, create_inputs and create_tests_outputs
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mydatapreprocessing-1.0.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7077dc7b967ddb91ef1eeccb51a103842d8f223793f4cb2497cc971d364742cb |
|
MD5 | 48c1d884e0445a07710b1bbc09ca9fc0 |
|
BLAKE2b-256 | 2daf940c5e7889294190871b93a626b57f14586885aaa4f304f4502cae33d83e |
Hashes for mydatapreprocessing-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 413d4d9f69e64ab916302c3002eaac3fd24d6f59c01d45ccf3230e6e4f38a1c0 |
|
MD5 | de0c59cf6ccf2151118735c1f267cf64 |
|
BLAKE2b-256 | 51d2317128d24bb7b4178aed9dc9c3f3b2bc450e2ef3b409c77d91f8e158f7db |