pangoro 0.33

pangoro python package

Project description

Pangoro: powerful Python data cleaning toolkit

What is it?

Pangoro is a Python package that provides fast and flexible methods for cleaning numerical and categorical features in a dataframe. It aims to be a fundamental tool for doing data wrangling in Python.

Main Features:

• For numerical features, pangoro provides the following tools:
  • Handle NA, drop, replace with mean, replace with mode, lamda function, replace with a number, replace with min replace with max, Use KNN classifications. Keep
  • Handle outliers, Keep, Percentile
  • Scaling, standard or min max
  • Convert to nuemerical
  • Scan and apply to all numerical or supply a list of features
• For categorical nominal features, pangoro provides the following tools:
  • Handle NA, Drop, replace with mode, use KNN classification for imputation.
  • Replace with sequence numbers based on supplied dictionary or based on alphabetical order.
  • Scan and apply to all Categorical or supply a list of features

Where to get it:

The source code is currently hosted on GitHub
Binary installers for the latest released version are available at the Python Package Index PyPi

pip install pangoro

Usage

1. Import

Import what you need from the pangoro package. The choices are: PangoroDataFrame, a class for cleaning numerical and categorical features in a dataframe in addition to plotting features correlations.

For this demonstration, we will import:

>>> from pangoro.preprocessing import PangoroDataFrame

For these examples, we'll also use pandas to :

>>> import pandas as pd

2. Load some Data

Typicall data will is read from a tabular formatted file, but for illustration we'll create a simple dataframe from a Python dict:

>>> sample_data = pd.DataFrame({'pet':      ['cat', 'dog', 'dog', pd.NA, 'cat', 'dog', 'cat', 'fish'],
                       'color': ['white', 'brown', 'black', 'gold', pd.NA,'black', 'white', 'silver'],
                       'weight':   [50.5, 22.5, 4, pd.NA , 12, 39, 16, pd.NA]})
>>> sample_data['weight'] = pd.to_numeric(sample_data['weight'])
>>> sample_data.dtypes

3. Create Instance and Apply Transformation

3.1 We will start by creating an instance of PangoroDataFrame using the created sample_data DataFrame above:

>>> pdf = PangoroDataFrame(sample_data)

3.2 Then, we will automate cleaning of numerical features in PangoroDataFrame by replacing NAs with mean and without performing any scaling:

>>> pdf.numerical_transformation(col = ['weight'], na_treat = 'mean', out_treat = True, out_upper = 0.8,
                         out_lower = 0.2, scaling='no', knn_neighbors = 0)

Similarly, as we continue with this example and using pdf as an instance PangoroDataFrame we can apply the other functions to clean and preprocess categorical ordinal by calling pdf.categorical_ordinal_transformation or categorical nominal by calling pdf.categorical_ordinal_transformation

In addition, we can quickly plot correlations between features by calling: pdf.plot_all_correlations

4. Getting Help

To learn more about we can issue the following command:

>>> help(PangoroDataFrame)

As a result we get detailed information about the package and each function included.

License

MIT

Background

Work on pangoro started in 2022 by a group of IE University students and has been under active development since then.

Contributing to pangoro

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome. Please contact us on GitHub