team-3-library 0.0.1

Data set cleaning library

DATASET CLEANING LIBRARY

Table of Contents

• General Info
• Functions of the library
• Features
• Setup
• Project Status
• Room for Improvement
• License

General Information

This code identifies null percentages per column, removes columns with nulls exceeding a specified threshold and fills missing values. It also analyzes column distribution using skewness and can show a graph to see how the distribution looks like.

The purpose of the project is to optimize the process of cleaning dataset as it minimizes data loss and enchances data quality as well as gives the opportunity to decide how to fill the missing values.

This project is undertaken in order to create a reliable data cleaning library that standardizes the preparation or preprocessing of datasets. It has an educational value since it is done to understand data processing practical applications.

Functions of the library

• analyze_nulls(self): This function calculates the missing values percentage per column.

• drop_null_columns(self, percentage): It compares the percentage given and depending on that it eliminates the columns that have greater percentage of null values than the given percentage.

• fill_nulls(self, column, e): Fills the null values with mean/median/mode/random value for numerical data and mode/ffill/bfill/random for categorical data. Also, it has the option of "auto". It is a value, which if the column has numerical data, depending on the distribution it fills the values with the mean/median/mode and in categorical data it fills with the mode.

• analyze_distribution(self, column): This function analyzes the distribution of the column data.

• clean_column_customized(self, column): It cleans a column in the dataset based on customized rules

• graph_num(self, column): It shows an histogram of the data column.

• graph_obj(self, column): It shows an countplot of the data column.

• clean_dataset_percentage(self, percentage): Eliminates the columns and cleans the rest of the columns.

• percentage(self): It requests for a percentage.

Features

The ready features:

• Awesome feature 1: It automatically analyzes to determine how to fill the missing values of each data column.
• Awesome feature 2: It analyzes the distribution of numerical data using Pearson's skewness coefficient.
• Awesome feature 3: It is interactive. Allows the user to choose how missing values will be replaced.

Setup

This library requires:

Python

Pandas: pip install pandas

Numpy: pip install numpy

Seaborn: pip install seaborn

Matplotlib.pyplot: pip install matplotlib

Project Status

Project is: Complete

Room for Improvement

This project can be improved in many aspects.

Room for improvement:

• Improvement to be done 1: It could ask for the percentage for each column instead of using the same threshold for all columns.
• Improvement to be done 2: Show better graphs and more figures.

To do:

• Feature to be added 1: It could write the new dataset in another .csv.
• Feature to be added 2: It could be done as a telegram bot.

License

MIT