Cleans data, best to be used as a part of initial preprocessor

These details have not been verified by PyPI

Project description

refineryframe

Under construction! Not ready for use yet! Currently experimenting and planning!

Initial plans

Goal of the package is to simplify life for data scientists, that have to deal with imperfect raw data. The package suppose to detect and clean unexpected values, while doubling as safeguard in production code based on predifined conditions that arise from business assumptions or any other source. The package is well suited to be an initial preprocessing step in ml pipelines situated between data gathering and training/scoring steps.

Installation

Install refineryframe via pip with

pip install refineryframe

Package usage example

import os 
import sys 
import numpy as np
import pandas as pd
import logging
sys.path.append(os.path.dirname(sys.path[0])) 
from refineryframe.refiner import Refiner

Creating example data (exceptionally messy dataframe)

df = pd.DataFrame({
    'num_id' : [1, 2, 3, 4, 5],
    'NumericColumn': [1, -np.inf, np.inf,np.nan, None],
    'NumericColumn_exepted': [1, -996, np.inf,np.nan, None],
    'NumericColumn2': [None, None, 1,None, None],
    'NumericColumn3': [1, 2, 3, 4, 5],
    'DateColumn': pd.date_range(start='2022-01-01', periods=5),
    'DateColumn2': [pd.NaT,pd.to_datetime('2022-01-01'),pd.NaT,pd.NaT,pd.NaT],
    'DateColumn3': ['2122-05-01',
                    '2022-01-01',
                    '2021-01-01',
                    '1000-01-09',
                    '1850-01-09'],
    'CharColumn': ['Fół', None, np.nan, 'nót eXpęćTęd', '']
})

df

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	num_id	NumericColumn	NumericColumn_exepted	NumericColumn2	NumericColumn3	DateColumn	DateColumn2	DateColumn3	CharColumn
0	1	1.0	1.0	NaN	1	2022-01-01	NaT	2122-05-01	Fół
1	2	-inf	-996.0	NaN	2	2022-01-02	2022-01-01	2022-01-01	None
2	3	inf	inf	1.0	3	2022-01-03	NaT	2021-01-01	NaN
3	4	NaN	NaN	NaN	4	2022-01-04	NaT	1000-01-09	nót eXpęćTęd
4	5	NaN	NaN	NaN	5	2022-01-05	NaT	1850-01-09

Defining specification for the dataframe

MISSING_TYPES = {'date_not_delivered': '1850-01-09',
                 'date_other_missing_type': '1850-01-08',
                 'numeric_not_delivered': -999,
                 'character_not_delivered': 'missing'}

unexpected_exceptions = {
    "col_names_types": "NONE",
    "missing_values": ["NumericColumn_exepted"],
    "missing_types": "NONE",
    "inf_values": "NONE",
    "date_format": "NONE",
    "duplicates": "ALL",
    "date_range": "NONE",
    "numeric_range": "NONE"
}

replace_dict = {-996 : -999,
                "1000-01-09": "1850-01-09"}

Initializing Refiner class

tns = Refiner(dataframe = df,
              replace_dict = replace_dict,
              loggerLvl = logging.DEBUG,
              unexpected_exceptions_duv = unexpected_exceptions)

function for detecting column types

tns.get_type_dict_from_dataframe()

{'num_id': 'int64',
 'NumericColumn': 'float64',
 'NumericColumn_exepted': 'float64',
 'NumericColumn2': 'float64',
 'NumericColumn3': 'int64',
 'DateColumn': 'datetime64[ns]',
 'DateColumn2': 'datetime64[ns]',
 'DateColumn3': 'object',
 'CharColumn': 'object'}

adding expected types

types_dict_str = {'num_id' : 'int64', 
                   'NumericColumn' : 'float64', 
                   'NumericColumn_exepted' : 'float64', 
                   'NumericColumn2' : 'float64', 
                   'NumericColumn3' : 'int64', 
                   'DateColumn' : 'datetime64[ns]', 
                   'DateColumn2' : 'datetime64[ns]', 
                   'DateColumn3' : 'datetime64[ns]', 
                   'CharColumn' : 'object'}

moulding types

tns.set_types(type_dict = types_dict_str)

tns.get_type_dict_from_dataframe()

{'num_id': 'int64',
 'NumericColumn': 'float64',
 'NumericColumn_exepted': 'float64',
 'NumericColumn2': 'float64',
 'NumericColumn3': 'int64',
 'DateColumn': 'datetime64[ns]',
 'DateColumn2': 'datetime64[ns]',
 'DateColumn3': 'datetime64[ns]',
 'CharColumn': 'object'}

Check independent conditions

tns.check_missing_types()
tns.check_missing_values()
tns.check_inf_values()
tns.check_col_names_types()
tns.check_date_format()
tns.check_duplicates()
tns.check_numeric_range()

WARNING:Refiner:Column NumericColumn_exepted: (-999) : 1 : 20.00%
WARNING:Refiner:Column DateColumn3: (1850-01-09) : 2 : 40.00%
WARNING:Refiner:Column NumericColumn: (NA) : 2 : 40.00%
WARNING:Refiner:Column NumericColumn_exepted: (NA) : 2 : 40.00%
WARNING:Refiner:Column NumericColumn2: (NA) : 4 : 80.00%
WARNING:Refiner:Column DateColumn2: (NA) : 4 : 80.00%
WARNING:Refiner:Column CharColumn: (NA) : 2 : 40.00%
WARNING:Refiner:Column NumericColumn: (INF) : 2 : 40.00%
WARNING:Refiner:Column NumericColumn_exepted: (INF) : 1 : 20.00%
WARNING:Refiner:Column DateColumn2 has non-date values or unexpected format.

Using the main function to detect unexpected values

tns.detect_unexpected_values(earliest_date = "1920-01-01",
                         latest_date = "DateColumn3")

DEBUG:Refiner:=== checking column names and types
DEBUG:Refiner:=== checking for presence of missing values
WARNING:Refiner:Column CharColumn: (NA) : 2 : 40.00%
WARNING:Refiner:Column DateColumn2: (NA) : 4 : 80.00%
WARNING:Refiner:Column NumericColumn: (NA) : 2 : 40.00%
WARNING:Refiner:Column NumericColumn2: (NA) : 4 : 80.00%
DEBUG:Refiner:=== checking for presence of missing types
WARNING:Refiner:Column DateColumn3: (1850-01-09) : 2 : 40.00%
WARNING:Refiner:Column NumericColumn_exepted: (-999) : 1 : 20.00%
DEBUG:Refiner:=== checking propper date format
WARNING:Refiner:Column DateColumn2 has non-date values or unexpected format.
DEBUG:Refiner:=== checking expected date range
WARNING:Refiner:** Not all dates in DateColumn are later than DateColumn3
WARNING:Refiner:Column DateColumn : future date : 4 : 80.00%
DEBUG:Refiner:=== checking for presense of inf values in numeric colums
WARNING:Refiner:Column NumericColumn: (INF) : 2 : 40.00%
WARNING:Refiner:Column NumericColumn_exepted: (INF) : 1 : 20.00%
DEBUG:Refiner:=== checking expected numeric range
WARNING:Refiner:Percentage of passed tests: 50.00%

tns.duv_score

0.5

Using function to replace unexpected values with missing types

tns.replace_unexpected_values(numeric_lower_bound = "NumericColumn3",
                                numeric_upper_bound = 4,
                                earliest_date = "1920-01-02",
                                latest_date = "DateColumn2",
                                unexpected_exceptions = {"irregular_values": "NONE",
                                                            "date_range": "DateColumn",
                                                            "numeric_range": "NONE",
                                                            "capitalization": "NONE",
                                                            "unicode_character": "NONE"})

DEBUG:Refiner:=== replacing missing values in category cols with missing types
DEBUG:Refiner:=== replacing all upper case characters with lower case
DEBUG:Refiner:=== replacing character unicode to latin
DEBUG:Refiner:=== replacing missing values in date cols with missing types
DEBUG:Refiner:=== replacing missing values in numeric cols with missing types
DEBUG:Refiner:=== replacing values outside of expected date range
DEBUG:Refiner:=== replacing values outside of expected numeric range
DEBUG:Refiner:** Usable values in the dataframe:  44.44%
DEBUG:Refiner:** Uncorrected data quality score:  32.22%
DEBUG:Refiner:** Corrected data quality score:  52.57%

tns.dataframe

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	num_id	NumericColumn	NumericColumn_exepted	NumericColumn2	NumericColumn3	DateColumn	DateColumn2	DateColumn3	CharColumn
0	1	1.0	1.0	-999.0	1	2022-01-01	1850-01-09	1850-01-09	fol
1	2	-999.0	-999.0	-999.0	2	2022-01-02	2022-01-01	2022-01-01	missing
2	3	-999.0	-999.0	-999.0	3	2022-01-03	1850-01-09	1850-01-09	missing
3	4	-999.0	-999.0	-999.0	4	2022-01-04	1850-01-09	1850-01-09	not expected
4	5	-999.0	-999.0	-999.0	-999	2022-01-05	1850-01-09	1850-01-09	missing

Use complex targeted conditions

unexpected_conditions = {
    '1': {
        'description': 'Replace numeric missing with with zero',
        'group': 'regex_columns',
        'features': r'^Numeric',
        'query': "{col} < 0",
        'warning': True,
        'set': 0
    },
    '2': {
        'description': "Clean text column from '-ing' endings and 'not ' beginings",
        'group': 'regex clean',
        'features': ['CharColumn'],
        'query': [r'ing', r'^not.'],
        'warning': False,
        'set': ''
    },
    '3': {
        'description': "Detect/Replace numeric values in certain column with zeros if > 2",
        'group': 'multicol mapping',
        'features': ['NumericColumn3'],
        'query': '{col} > 2',
        'warning': True,
        'set': 0
    },
    '4': {
        'description': "Replace strings with values if some part of the string is detected",
        'group': 'string check',
        'features': ['CharColumn'],
        'query': f"CharColumn.str.contains('cted', regex = True)",
        'warning': False,
        'set': 'miss'
    }
    }

- to detect unexpected values

tns.detect_unexpected_values(unexpected_conditions = unexpected_conditions)

DEBUG:Refiner:=== checking column names and types
DEBUG:Refiner:=== checking for presence of missing values
DEBUG:Refiner:=== checking for presence of missing types
WARNING:Refiner:Column CharColumn: (missing) : 3 : 60.00%
WARNING:Refiner:Column DateColumn2: (1850-01-09) : 4 : 80.00%
WARNING:Refiner:Column DateColumn3: (1850-01-09) : 4 : 80.00%
WARNING:Refiner:Column NumericColumn: (-999) : 4 : 80.00%
WARNING:Refiner:Column NumericColumn_exepted: (-999) : 4 : 80.00%
WARNING:Refiner:Column NumericColumn2: (-999) : 5 : 100.00%
WARNING:Refiner:Column NumericColumn3: (-999) : 1 : 20.00%
DEBUG:Refiner:=== checking propper date format
DEBUG:Refiner:=== checking expected date range
DEBUG:Refiner:=== checking for presense of inf values in numeric colums
DEBUG:Refiner:=== checking expected numeric range
DEBUG:Refiner:=== checking additional cons
DEBUG:Refiner:Replace numeric missing with with zero
WARNING:Refiner:Replace numeric missing with with zero :: 1
DEBUG:Refiner:Detect/Replace numeric values in certain column with zeros if > 2
WARNING:Refiner:Detect/Replace numeric values in certain column with zeros if > 2 :: 2
WARNING:Refiner:Percentage of passed tests: 75.00%

- to replace unecpected values

tns.replace_unexpected_values(unexpected_conditions = unexpected_conditions)

DEBUG:Refiner:=== replacing missing values in category cols with missing types
DEBUG:Refiner:=== replacing all upper case characters with lower case
DEBUG:Refiner:=== replacing character unicode to latin
DEBUG:Refiner:=== replacing with additional cons
DEBUG:Refiner:Replace numeric missing with with zero
DEBUG:Refiner:Clean text column from '-ing' endings and 'not ' beginings
DEBUG:Refiner:Detect/Replace numeric values in certain column with zeros if > 2
DEBUG:Refiner:Replace strings with values if some part of the string is detected
DEBUG:Refiner:=== replacing missing values in date cols with missing types
DEBUG:Refiner:=== replacing missing values in numeric cols with missing types
DEBUG:Refiner:=== replacing values outside of expected date range
DEBUG:Refiner:=== replacing values outside of expected numeric range
DEBUG:Refiner:** Usable values in the dataframe:  82.22%
DEBUG:Refiner:** Uncorrected data quality score:  88.89%
DEBUG:Refiner:** Corrected data quality score:  97.53%

tns.dataframe

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	num_id	NumericColumn	NumericColumn_exepted	NumericColumn3	DateColumn	DateColumn2	DateColumn3	CharColumn
0	1	1.0	1.0	1	2022-01-01	1850-01-09	1850-01-09	fol
1	2	0.0	0.0	2	2022-01-02	2022-01-01	2022-01-01	miss
2	3	0.0	0.0	0	2022-01-03	1850-01-09	1850-01-09	miss
3	4	0.0	0.0	0	2022-01-04	1850-01-09	1850-01-09	miss
4	5	0.0	0.0	0	2022-01-05	1850-01-09	1850-01-09	miss

tns.detect_unexpected_values(unexpected_exceptions = {
    "col_names_types": "NONE",
    "missing_values": "NONE",
    "missing_types": "ALL",
    "inf_values": "NONE",
    "date_format": "NONE",
    "duplicates": "ALL",
    "date_range": "NONE",
    "numeric_range": "NONE"
})

DEBUG:Refiner:=== checking column names and types
DEBUG:Refiner:=== checking for presence of missing values
DEBUG:Refiner:=== checking propper date format
DEBUG:Refiner:=== checking expected date range
DEBUG:Refiner:=== checking for presense of inf values in numeric colums
DEBUG:Refiner:=== checking expected numeric range

Scores

print(f'duv_score: {tns.duv_score :.4}')
print(f'ruv_score0: {tns.ruv_score0 :.4}')
print(f'ruv_score1: {tns.ruv_score1 :.4}')
print(f'ruv_score2: {tns.ruv_score2 :.4}')

duv_score: 1.0
ruv_score0: 0.8222
ruv_score1: 0.8889
ruv_score2: 0.9753

Project details

These details have not been verified by PyPI

Development Status
- 1 - Planning
Intended Audience
- Developers
Operating System
Programming Language
- Python :: 3

Release history Release notifications | RSS feed

0.2.2

Sep 5, 2023

0.2.1

Sep 4, 2023

0.1.28

Sep 4, 2023

0.1.27

Sep 4, 2023

0.1.26

Sep 4, 2023

0.1.25

Sep 4, 2023

0.1.24

Aug 18, 2023

0.1.23

Aug 18, 2023

0.1.22

Aug 15, 2023

0.1.21

Aug 15, 2023

0.1.20

Aug 15, 2023

0.1.19

Aug 14, 2023

0.1.18

Aug 13, 2023

0.1.17

Aug 13, 2023

0.1.16

Aug 10, 2023

0.1.15

Aug 9, 2023

0.1.14

Aug 6, 2023

0.1.13

Aug 5, 2023

0.1.12

Aug 5, 2023

0.1.11

Aug 5, 2023

0.1.10

Aug 5, 2023

0.1.9

Aug 5, 2023

0.1.8

Aug 5, 2023

0.1.7

Aug 5, 2023

0.1.6

Aug 5, 2023

0.1.5

Aug 5, 2023

0.1.4

Aug 5, 2023

0.1.3

Aug 3, 2023

0.1.2

Aug 2, 2023

0.1.1

Aug 1, 2023

0.0.10

Jul 31, 2023

0.0.9

Jul 30, 2023

0.0.8

Jul 30, 2023

0.0.7

Jul 30, 2023

This version

0.0.6

Jul 29, 2023

0.0.5

Jul 26, 2023

0.0.4

Jul 25, 2023

0.0.3

Jul 25, 2023

0.0.2

Jul 25, 2023

0.0.1

Jul 25, 2023

0.0.0

Jul 30, 2023

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

refineryframe-0.0.6.tar.gz (32.1 kB view hashes)

Uploaded Jul 29, 2023 Source

Built Distribution

refineryframe-0.0.6-py3-none-any.whl (32.1 kB view hashes)

Uploaded Jul 29, 2023 Python 3

Hashes for refineryframe-0.0.6.tar.gz

Hashes for refineryframe-0.0.6.tar.gz
Algorithm	Hash digest
SHA256	`87a13c7ff7990feb3366cfe37351d2cf09c50e504356702d621020ede84c045e`
MD5	`6074d3c05ffc56807055c369e8730be6`
BLAKE2b-256	`72c16d1b12c67ec7d40a82d3d05a3b7e75f656303cd777ff200c0cb98dd818b4`

Hashes for refineryframe-0.0.6-py3-none-any.whl

Hashes for refineryframe-0.0.6-py3-none-any.whl
Algorithm	Hash digest
SHA256	`0066ec59c46a8a5bbd12b95771cb3e49ec8776036b7cceb2c0e2c2cfc41d5c55`
MD5	`f04304f513779b494036ea8da7b74311`
BLAKE2b-256	`49a8b7405abba129a864f08eb77009664e05f5bf2c5ca228c559cc5767e2bfc0`