Python package for Outlier Removal Algorithm using z_score or iqr.
Project description
outlier-python
Package Description :
Python package for Outlier Removal Algorithm using z_score or iqr.
Motivation :
This is a part of project - II made for UCS633 - Data analytics and visualization at TIET.
@Author : Sourav Kumar
@Roll no. : 101883068
Algorithm :
-
Z-SCORE : If the population mean and population standard deviation are known, the standard score of a raw score x is calculated as:
z = (x - mean) / std.
mean : is the mean of the sample.
std : is the standard deviation of the sample. -
Interquartile range : interquartile range (IQR), also called the midspread, middle 50%, or H‑spread, is a measure of statistical dispersion, being equal to the difference between 75th and 25th percentiles, or between upper and lower quartiles.
IQR = Q3 − Q1
The IQR of a set of values is calculated as the difference between the upper and lower quartiles, Q3 and Q1. Each quartile is a median calculated as follows :
Given an even 2n or odd 2n+1 number of values.
first quartile Q1 = median of the n smallest values
third quartile Q3 = median of the n largest values
The second quartile Q2 is the same as the ordinary median.
Getting started Locally :
Run On Terminal
python -m outlier.outlier inputFilePath outputFilePath z_score
orpython -m outlier.outlier inputFilePath outputFilePath iqr
ex. python -m outlier outlier C:/Users/DELL/Desktop/train.csv C:/Users/DELL/Desktop/output.csv z_score
Run In IDLE
from outlier import outlier
o = outlier.outlier(inputFilePath, outputFilePath)
o.outlier_main('z_score')
or
o.outlier_main('iqr')
Run on Jupyter
Open terminal (cmd)
jupyter notebook
Create a new python3 file.
from outlier import outlier
o = outlier.outlier(inputFilePath, outputFilePath)
o.outlier_main('z_score')
or
o.outlier_main('iqr')
- NOTE :
outlier_main()
doesn't necessarily require anymethod
argument , if no argument is provided, it usesz_score
by default as the algorithm for removal of outliers from the dataset. - The algorithm only reports missing data containing columns and not handles them, it assumes that it has been handled already.
Also in case of z-score method, it will not affect much, but it may be possible to give wrong output in case of IQR if missing values are found.
OUTPUT :
Removes all the valid rows contaning outlier values from the dataset and prints the number of rows removed along with the columns which were considered for the algorithm.
Also , the final dataframe will be written to the output file path you provided.
TESTING :
- The package has been extensively tested on various datasets consisting varied types of expected and unexpected input data and any preprocessing , if required has been taken care of.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for outlier-python-souravdlboy-1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4bd11547b98f961a7d0dbdbf297792a2130eca3546a87cf3ca1a18444a8ea2ba |
|
MD5 | 0f7bb1b09296b3cd6025c650c5e7b80b |
|
BLAKE2b-256 | 210a87ba6e26aee0561a28a2277f7398b265d8afaa3dd3bde1ef419208f1f4e8 |
Hashes for outlier_python_souravdlboy-1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ad62a135229326f2abf96499cfe697378281ea5e449a092e8182a3c8fcb1843 |
|
MD5 | d432e3d2229ab69ec304ed21e044d17a |
|
BLAKE2b-256 | e422b5e8f0c8dbe00239ce7acc07e24b50ee74a4e916d4f1b149a89392cda9e1 |