Python package for Outlier Removal Algorithm using z_score or iqr.
Project description
outlier-python
Package Description :
Python package for Outlier Removal Algorithm using z_score or iqr.
Motivation :
This is a part of project - II made for UCS633 - Data analytics and visualization at TIET.
@Author : Sourav Kumar
@Roll no. : 101883068
Algorithm :
-
Z-SCORE : If the population mean and population standard deviation are known, the standard score of a raw score x is calculated as:
z = (x - mean) / std.
mean : is the mean of the sample.
std : is the standard deviation of the sample. -
Interquartile range : interquartile range (IQR), also called the midspread, middle 50%, or H‑spread, is a measure of statistical dispersion, being equal to the difference between 75th and 25th percentiles, or between upper and lower quartiles.
IQR = Q3 − Q1
The IQR of a set of values is calculated as the difference between the upper and lower quartiles, Q3 and Q1. Each quartile is a median calculated as follows :
Given an even 2n or odd 2n+1 number of values.
first quartile Q1 = median of the n smallest values
third quartile Q3 = median of the n largest values
The second quartile Q2 is the same as the ordinary median.
Getting started Locally :
Run On Terminal
python -m outlier.outlier inputFilePath outputFilePath z_score
orpython -m outlier.outlier inputFilePath outputFilePath iqr
ex. python -m outlier outlier C:/Users/DELL/Desktop/train.csv C:/Users/DELL/Desktop/output.csv z_score
Run In IDLE
from outlier import outlier
o = outlier.outlier(inputFilePath, outputFilePath)
o.outlier_main('z_score')
or
o.outlier_main('iqr')
Run on Jupyter
Open terminal (cmd)
jupyter notebook
Create a new python3 file.
from outlier import outlier
o = outlier.outlier(inputFilePath, outputFilePath)
o.outlier_main('z_score')
or
o.outlier_main('iqr')
- NOTE :
outlier_main()
doesn't necessarily require anymethod
argument , if no argument is provided, it usesz_score
by default as the algorithm for removal of outliers from the dataset. - The algorithm only reports missing data containing columns and not handles them, it assumes that it has been handled already.
Also in case of z-score method, it will not affect much, but it may be possible to give wrong output in case of IQR if missing values are found.
OUTPUT :
Removes all the valid rows contaning outlier values from the dataset and prints the number of rows removed along with the columns which were considered for the algorithm.
Also , the final dataframe will be written to the output file path you provided.
TESTING :
- The package has been extensively tested on various datasets consisting varied types of expected and unexpected input data and any preprocessing , if required has been taken care of.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for outlier-python-souravdlboy-1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab3a5fa52bb48f9c936d57a3cac212e22ea5e65caee72f96510d3cd6eafc061f |
|
MD5 | 392ffc76e75e10de7f2f0e1dcffe6717 |
|
BLAKE2b-256 | dfcfbac5a2718e32137d5d213318cedf683a50af1c8a16ef3bff3ae62d5b4428 |
Hashes for outlier_python_souravdlboy-1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ee9fd624017771e74e9e09c3d0170753bd1ade650b459d46980f8ef51f7e11d |
|
MD5 | 4c718e9d57b5bde11da48b60b19f5a9f |
|
BLAKE2b-256 | a2a26e0e60434e2c19a7fe4d79705d71c397192c4de59d809c6707f57444d277 |