pangoro 0.35

pangoro python package

Project description

Pangoro: powerful Python data cleaning toolkit

What is it?

Pangoro is a Python package that provides fast and flexible methods for cleaning numerical and categorical features in a dataframe. It aims to be a fundamental tool for doing data wrangling in Python.

Main Features:

• For numerical features, pangoro provides the following tools:
  • Handle NA, drop, replace with mean, replace with mode, lamda function, replace with a number, replace with min replace with max, Use KNN classifications. Keep
  • Handle outliers, Keep, Percentile
  • Scaling, standard or min max
  • Convert to nuemerical
  • Scan and apply to all numerical or supply a list of features
• For categorical nominal features, pangoro provides the following tools:
  • Handle NA, Drop, replace with mode, use KNN classification for imputation.
  • Replace with sequence numbers based on supplied dictionary or based on alphabetical order.
  • Scan and apply to all Categorical or supply a list of features

Where to get it:

The source code is currently hosted on GitHub
Binary installers for the latest released version are available at the Python Package Index PyPi

pip install --upgrade pangoro

Usage

1. Import

Import what you need from the pangoro package. The choices are: PangoroDataFrame, a class for cleaning numerical and categorical features in a dataframe in addition to plotting features correlations.

For this demonstration, we will import:

>>> from pangoro.preprocessing import PangoroDataFrame

For these examples, we'll also use pandas to :

>>> import pandas as pd

2. Load some Data

Typicall data will is read from a tabular formatted file, but for illustration we'll create a simple dataframe from a Python dict:

>>> sample_data = pd.DataFrame({'pet':      ['cat', 'dog', 'dog', pd.NA, 'cat', 'dog', 'cat', 'fish'],
                       'color': ['white', 'brown', 'black', 'gold', pd.NA,'black', 'white', 'silver'],
                       'weight':   [50.5, 22.5, 4, pd.NA , 12, 39, 16, pd.NA]})
>>> sample_data['weight'] = pd.to_numeric(sample_data['weight'])
>>> sample_data.dtypes

3. Create Instance and Apply Transformation

3.1 We will start by creating an instance of PangoroDataFrame using the created sample_data DataFrame above:

>>> pdf = PangoroDataFrame(sample_data)

3.2 Then, we will automate cleaning of numerical features in PangoroDataFrame by replacing NAs with mean and without performing any scaling:

>>> pdf.numerical_transformation(col = ['weight'], na_treat = 'mean', out_treat = True, out_upper = 0.8,
                         out_lower = 0.2, scaling='no', knn_neighbors = 0)

Similarly, as we continue with this example and using pdf as an instance PangoroDataFrame we can apply the other functions to clean and preprocess categorical ordinal by calling pdf.categorical_ordinal_transformation or categorical nominal by calling pdf.categorical_nominal_transformation

In addition, we can quickly plot correlations between features by calling: pdf.plot_all_correlations

---

NOTE PangoroDataFrame object has to be re-assigned to itself in order to store changes inplace after transforming nominal categories by calling categorical_nominal_transformation, which will result in adding new features to the original PangoroDataFrame, for example:

>>> pdf =pdf.categorical_nominal_transformation(col=['weight'],transform=True )

---

4. Getting Help

To learn more about we can issue the following command:

>>> help(PangoroDataFrame)

As a result we get detailed information about the package and each function included.

5. Unit Test

pangoro package is equipped with test module that contains TestDataFrame class to perform all unit tests required to validate output of the PangoroDataFrame functions. Tests can be called by the follwoing command from the main pangoro package folder:

>>> python -m tests.test

As a result, all test functions will be called and performed and the output will be the number of performed tests along with results of the tests (i.e. OK or FAILED(errors))

License

MIT

Background

Work on pangoro started in 2022 by a group of IE University students and has been under active development since then.

Contributing to pangoro

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome. Please contact us on GitHub