Skip to main content

Python package for Outlier Removal Algorithm using z_score or iqr.

Project description

outlier-python

Package Description :

Python package for Outlier Removal Algorithm using z_score or iqr.

Motivation :

This is a part of project - II made for UCS633 - Data analytics and visualization at TIET.
@Author : Sourav Kumar
@Roll no. : 101883068

Algorithm :

  • Z-SCORE : If the population mean and population standard deviation are known, the standard score of a raw score x is calculated as:
    z = (x - mean) / std.
    mean : is the mean of the sample.
    std : is the standard deviation of the sample.

  • Interquartile range : interquartile range (IQR), also called the midspread, middle 50%, or H‑spread, is a measure of statistical dispersion, being equal to the difference between 75th and 25th percentiles, or between upper and lower quartiles.
    IQR = Q3 − Q1
    The IQR of a set of values is calculated as the difference between the upper and lower quartiles, Q3 and Q1. Each quartile is a median calculated as follows :
    Given an even 2n or odd 2n+1 number of values.
    first quartile Q1 = median of the n smallest values
    third quartile Q3 = median of the n largest values
    The second quartile Q2 is the same as the ordinary median.

Getting started Locally :

Run On Terminal
python -m outlier.outlier inputFilePath outputFilePath z_score
or python -m outlier.outlier inputFilePath outputFilePath iqr
ex. python -m outlier outlier C:/Users/DELL/Desktop/train.csv C:/Users/DELL/Desktop/output.csv z_score

Run In IDLE
from outlier import outlier
o = outlier.outlier(inputFilePath, outputFilePath)
o.outlier_main('z_score') or
o.outlier_main('iqr')

Run on Jupyter
Open terminal (cmd)
jupyter notebook
Create a new python3 file.
from outlier import outlier
o = outlier.outlier(inputFilePath, outputFilePath) o.outlier_main('z_score') or
o.outlier_main('iqr')

  • NOTE : outlier_main() doesn't necessarily require any method argument , if no argument is provided, it uses z_score by default as the algorithm for removal of outliers from the dataset.
  • The algorithm only reports missing data containing columns and not handles them, it assumes that it has been handled already.
    Also in case of z-score method, it will not affect much, but it may be possible to give wrong output in case of IQR if missing values are found.

OUTPUT :

Removes all the valid rows contaning outlier values from the dataset and prints the number of rows removed along with the columns which were considered for the algorithm.
Also , the final dataframe will be written to the output file path you provided.

output result on jupyter output result on idle output result on cmd

TESTING :

  • The package has been extensively tested on various datasets consisting varied types of expected and unexpected input data and any preprocessing , if required has been taken care of.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

outlier-python-souravdlboy-1.1.tar.gz (4.6 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file outlier-python-souravdlboy-1.1.tar.gz.

File metadata

  • Download URL: outlier-python-souravdlboy-1.1.tar.gz
  • Upload date:
  • Size: 4.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.7.0b5

File hashes

Hashes for outlier-python-souravdlboy-1.1.tar.gz
Algorithm Hash digest
SHA256 ab3a5fa52bb48f9c936d57a3cac212e22ea5e65caee72f96510d3cd6eafc061f
MD5 392ffc76e75e10de7f2f0e1dcffe6717
BLAKE2b-256 dfcfbac5a2718e32137d5d213318cedf683a50af1c8a16ef3bff3ae62d5b4428

See more details on using hashes here.

File details

Details for the file outlier_python_souravdlboy-1.1-py3-none-any.whl.

File metadata

  • Download URL: outlier_python_souravdlboy-1.1-py3-none-any.whl
  • Upload date:
  • Size: 5.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.7.0b5

File hashes

Hashes for outlier_python_souravdlboy-1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 2ee9fd624017771e74e9e09c3d0170753bd1ade650b459d46980f8ef51f7e11d
MD5 4c718e9d57b5bde11da48b60b19f5a9f
BLAKE2b-256 a2a26e0e60434e2c19a7fe4d79705d71c397192c4de59d809c6707f57444d277

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page