Python package that ease the pain in preprocessing like outlier finding, numerical/categorical data and etc.
Project description
mealprep
Mealprep offers a toolkit, made with care, to help users save time in the data preprocessing kitchen.
Overview
Recognizing that the preparation step of a data science project often
requires the most time and effort, mealprep
aims to help data science
chefs of all specialties master their recipes of analysis. This package
tackles pesky tasks such as classifying columns as categorical or
numeric ingredients, straining NA values and outliers, and automating a
preprocessing recipe pipeline.
Functions
find_fruits_veg()
: This function will drop rows with NAs and find the
indices of columns with all numeric values or categorical values based
on the specification.
find_missing_ingredients()
: For each column with missing values, this
function will create a reference list of row indices, sum the number,
and calculate the proportion of missing values.
find_bad_apples()
: This function uses a univariate approach to outlier
detection. For each column with outliers (values that are 2 or more
standard deviations from the mean), this function will create a
reference list of row indices with outliers, and the total number of
outliers in that column.
make_recipe()
: This function is used to quickly apply the following
common data preprocessing techniques with one line of code: split the
dataset into a training set and testing set, apply standard scaling to
numeric features, apply onehotencoding to categorical features, fit
and transform training data, and fit testing data.
Mealprep and Python’s Ecosystem
mealprep complements many of the existing packages in the Python
ecosystem around the theme of data preprocessing. When preparing a
dataframe for a machine learning preprocessing pipeline, it is time
consuming to manually note which columns are categorical and numerical,
particularly for large datasets. The
pandas function df.select_dtypes()
comes close by allowing users to select columns with data corresponding
to specific data types however the output of this function is a pandas
dataframe. find_fruits_veg()
aims to fill this void by producing a
list of columns corresponding to the categorical and numerical groups.
In terms of missing values, pandas
package’s isna()
function converts all elements of a pandas.dataframe
or pandas.series to boolean values representing if they are missing
values. The package
autoimpute provides a
suite of tools to fill missing values in a dataset through multiple
univariate, multivariate and time series methods. The gap between these
packages is that neither provides you a summary of the missing values
including the list of indices where they occur.
find_missing_ingredients()
augments these tools by providing a summary
dataframe detailing which columns have missing values, as well as their
count and proportion.
The pandas package’s describe()
function is a staple in the data wrangling process because it returns
several summary statistics for each numeric column in a dataframe, such
as the mean, standard deviation, minimum, and maximum. Viewing these
statistics together is helpful for detecting outliers. However, the
output of this function does not tell you which rows of data these
outliers are found in, or how many outliers are present in the
dataframe. Packages like the
PyOD toolkit and other
functions that use clustering methods consider all variables at once to
detect outliers for multivariate data.
PyOD provides over 20
algorithms to select from in detecting these outliers, which is handy
for large multivariate datasets where you know you want to consider all
features in detecting outliers, but can be a bit extreme for initial
data exploration. The mealprep find_bad_apples()
function lives
happily in the space between pandas
and PyODtype solutions for
outlier detection, where it provides more information than the
pandas describe()
function to
point out datapoints which need further investigation, but does not
consider all variables at once like the
PyODtype functions do.
Lastly, there are many great tools in the data science ecosystem for
preprocessing data such as scikitlearn
preprocessing
in Python. However, you may find yourself frequently writing the same
lengthy code for common preprocessing tasks (e.g scale numeric features
and one hot encode categorical features). preprocess_recipe()
provides
a shortcut function to apply your favourite recipes quickly to
preprocess data in one line of code.
Installation:
pip install i https://test.pypi.org/simple/ mealprep
Examples
find_fruits_veg()
Find the column indices for either numerical or categorical variables in
your dataframe with the find_fruits_veg()
function. The example below
shows how to use find_fruits_veg() to find the index of the
categorical column in a toy dataframe.
First, load the required packages.
from mealprep.mealprep import find_fruits_veg import pandas as pd
If you don’t already have a dataframe to work with, run this code to set
up a toy dataframe (df
) for testing.
df = pd.DataFrame({'col1': [1, 2], 'col2': ['a', 'b']}) df
## col1 col2
## 0 1 a
## 1 2 b
Then, apply the find_fruits_veg()
function to the dataframe.
find_fruits_veg(df, type_of_out = 'categ')
## [1]
find_missing_ingredients()
Before launching into a new data analysis, running the function
find_missing_ingredients()
on a dataframe of interest will produce a
report on each column with missing values.
First, load the required packages
from mealprep.mealprep import find_missing_ingredients import pandas as pd import numpy as np
If you don’t already have a dataframe to work with, run this code to set
up a toy dataframe (df
) for testing.
test1= {'column1': ['a', 'b', 'c', 'd'], 'column2': [1, 2, np.NaN, 3], 'column3': [np.NaN] * 4} df = pd.DataFrame(test1) df
## column1 column2 column3
## 0 a 1.0 NaN
## 1 b 2.0 NaN
## 2 c NaN NaN
## 3 d 3.0 NaN
Then, apply the find_missing_ingredients()
function to the dataframe.
find_missing_ingredients(df)
## Column name NaN count NaN proportion NaN indices
## 0 column2 1 25.0% [2]
## 1 column3 4 100.0% [0, 1, 2, 3]
find_bad_apples()
Find the outliers in your data by applying the find_bad_apples()
function to your dataframe.
First, load the required packages.
from mealprep.mealprep import find_bad_apples import pandas as pd
If you don’t already have a dataframe to work with, run this code to set
up a toy dataframe (df
) for testing.
df = pd.DataFrame({'A' : [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1], 'B' : [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,100, 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,100], 'C' : [1,1,1,1,1,19,1,1,1,1,1,1,1,1,19,1,1,1,1, 1,1,1,1,1,1,1,19,1,1,1,1,1,1,1,1]}) df
## A B C
## 0 1 1 1
## 1 1 1 1
## 2 1 1 1
## 3 1 1 1
## 4 1 1 1
## 5 1 1 19
## 6 1 1 1
## 7 1 1 1
## 8 1 1 1
## 9 1 1 1
## 10 1 1 1
## 11 1 1 1
## 12 1 1 1
## 13 1 1 1
## 14 1 1 19
## 15 1 1 1
## 16 1 1 1
## 17 1 100 1
## 18 1 1 1
## 19 1 1 1
## 20 1 1 1
## 21 1 1 1
## 22 1 1 1
## 23 1 1 1
## 24 1 1 1
## 25 1 1 1
## 26 1 1 19
## 27 1 1 1
## 28 1 1 1
## 29 1 1 1
## 30 1 1 1
## 31 1 1 1
## 32 1 1 1
## 33 1 1 1
## 34 1 100 1
Then, apply the find_bad_apples()
function to the dataframe.
find_bad_apples(df)
## Variable Indices Total Outliers
## 0 B [17, 34] 2
## 1 C [5, 14, 26] 3
make_recipe()
Do you find yourself constantly applying the same data preprocessing
techniques time and time again? make_recipe
can help by applying your
favourite preprocessing recipes in only a few lines of code.
Below make_recipe
applies the following common recipe in only one line
of code:
 Split data into training, validation, and testing
 Standardise and scale numeric features
 One hot encode categorical features
First, load the required packages.
from mealprep.mealprep import make_recipe import pandas as pd import numpy as np from vega_datasets import data
If you don’t already have a dataframe to work with, run this code to
load the classic mtcars
dataset for testing.
df = pd.read_json(data.cars.url).drop(columns=["Year"]) X = df.drop(columns=["Name"]) y = df[["Name"]] df.info()
## <class 'pandas.core.frame.DataFrame'>
## RangeIndex: 406 entries, 0 to 405
## Data columns (total 8 columns):
## # Column NonNull Count Dtype
##    
## 0 Name 406 nonnull object
## 1 Miles_per_Gallon 398 nonnull float64
## 2 Cylinders 406 nonnull int64
## 3 Displacement 406 nonnull float64
## 4 Horsepower 400 nonnull float64
## 5 Weight_in_lbs 406 nonnull int64
## 6 Acceleration 406 nonnull float64
## 7 Origin 406 nonnull object
## dtypes: float64(4), int64(2), object(2)
## memory usage: 25.5+ KB
Then, use make_recipe
to quickly apply split your data and apply your
favourite preprocessing techniques!
X_train, X_valid, X_test, y_train, y_valid, y_test = make_recipe( X=X, y=y, recipe="ohe_and_standard_scaler", splits_to_return="train_test") X_train.head()
## Miles_per_Gallon Cylinders Displacement ... x0_Europe x0_Japan x0_USA
## 0 0.564509 0.846151 0.910090 ... 0.0 0.0 1.0
## 1 0.883582 0.846151 0.910090 ... 0.0 0.0 1.0
## 2 1.126078 0.846151 0.815709 ... 0.0 1.0 0.0
## 3 1.094674 0.308177 0.524498 ... 0.0 0.0 1.0
## 4 0.794242 0.846151 0.995032 ... 1.0 0.0 0.0
##
## [5 rows x 9 columns]
Documentation
The official documentation is hosted on Read the Docs: https://mealprep.readthedocs.io/en/latest/
Credits
This package was created with Cookiecutter and the UBCMDS/cookiecutterubcmds project template, modified from the pyOpenSci/cookiecutterpyopensci project template and the audreyr/cookiecutterpypackage.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mealprep1.0.15py3noneany.whl
Algorithm  Hash digest  

SHA256  e71084c7a862297549c72c9dc28269e9a00af3045a2f6a5569d925f25cb4cdd7 

MD5  f67dbbfb91e35db3360581dcf5dd7ff9 

BLAKE2256  79ba933f6c7d9dd1b62207245b04459b3750bf3bfdcb8d6756602a96073ac42c 