pyspan 0.1.1

A Python package for efficient data cleaning and preprocessing with Pandas.

pyspan

'pyspan' is a Python package designed to facilitate data cleaning and preprocessing using Pandas. It provides various functions to handle missing values, detect outliers, spell check data, and more. Additionally, it includes a logging utility to keep track of function calls and their parameters.

Installation

To use pyspan, simply install the package using pip:

``bash pip install pyspan

Functions

1. handle_nulls(data: pd.DataFrame, columns: List[str], method: str, value: Optional[Union[str, float]] = None) -> pd.DataFrame Handles missing values in the specified columns of a DataFrame. Parameters: data: DataFrame with missing values. columns: List of column names to apply the fill operation. method: Strategy to use for imputing missing values ('mean', 'median', 'mode', 'interpolate', 'forward_fill', 'backward_fill'). value: Custom value to fill NaNs with (optional).

2. remove_duplicates(df: pd.DataFrame) -> pd.DataFrame Removes duplicate rows from a DataFrame. Parameters: df: DataFrame to remove duplicates from.

3. remove_columns(df: pd.DataFrame, columns: List[str]) -> pd.DataFrame Removes specified columns from a DataFrame. Parameters: df: DataFrame to remove columns from. columns: List of column names to remove.

4. auto_rename_columns(df: pd.DataFrame) -> pd.DataFrame Automatically renames columns to remove spaces and special characters. Parameters: df: DataFrame to rename columns in.

5. rename_dataframe_columns(df: pd.DataFrame, rename_dict: dict) -> pd.DataFrame Renames columns in a DataFrame using a provided dictionary mapping. Parameters: df: DataFrame to rename columns in. rename_dict: Dictionary mapping current column names to new column names.

6. change_dt(df: pd.DataFrame, columns: Union[str, List[str]], date_format: str = "%Y-%m-%d", time_format: str = "%H:%M:%S", from_timezone: Optional[str] = None, to_timezone: Optional[str] = None) -> pd.DataFrame Changes the format of date and time columns and handles timezone conversion. Parameters: df: DataFrame containing date/time columns to reformat. columns: Name(s) of the column(s) to be reformatted. date_format: Desired date format. time_format: Desired time format. from_timezone: Original timezone of the datetime column(s). to_timezone: Desired timezone for the datetime column(s).

7. detect_delimiter(series: pd.Series) -> str Detects the most common delimiter in a Series of strings. Parameters: series: Series containing strings to analyze.

8. split_column(df: pd.DataFrame, column_name: str, delimiter: str = None) -> pd.DataFrame Splits a single column into multiple columns based on a delimiter. Parameters: df: DataFrame containing the column to split. column_name: Name of the column to be split. delimiter: Delimiter to use for splitting (optional).

9. impute(data, by=None, value=None, columns=None) -> pd.DataFrame Handles missing values using specified strategy or custom value. Parameters: data: DataFrame or Series with missing values. by: Strategy for imputing missing values ('mean', 'median', 'mode', 'interpolate', 'forward_fill', 'backward_fill'). value: Custom value to fill NaNs with. columns: List of column names to apply the fill operation.

10. spell_check_dataframe(data: pd.DataFrame, dictionary='en_US', columns=None) -> dict Performs spell check on specified columns of a DataFrame. Parameters: data: DataFrame containing columns to spell check. dictionary: Dictionary to use for spell checking ('en_US', 'en_GB', 'en_AU', 'en_IE'). columns: List of column names to perform spell check on.

11. detect_invalid_dates(series: pd.Series) -> pd.Series Detects invalid date values in a Series. Parameters: series: Series to check for invalid dates.

12. detect_data_entry_errors(data, spellcheck_dict='en_US', date_columns=None, numeric_columns=None, text_columns=None) -> pd.DataFrame Detects and flags data entry errors including invalid dates and misspelled words. Parameters: data: DataFrame to analyze. spellcheck_dict: Dictionary to use for spell checking. date_columns: List of columns to check for invalid dates. numeric_columns: List of columns to check for numeric format errors. text_columns: List of text columns to perform spell checking on.

13. data_type_conversions(data, column=None) -> pd.DataFrame or pd.Series Recommends and applies data type conversions based on the analysis of each column's data. Parameters: data: DataFrame or Series to analyze. column: Specific column to analyze (optional).

14. detect_outliers(data, method='iqr', threshold=1.5, columns=None, handle_missing=True) -> pd.DataFrame Detects outliers in a dataset using specified method and threshold. Parameters: data: DataFrame or Series to analyze. method: Outlier detection method ('z-score', 'iqr'). threshold: Threshold for outlier detection. columns: List of columns to apply the outlier detection on (optional). handle_missing: Whether to handle missing values by dropping them or not.

Logging Functions

1. display_logs() Prints stored log entries.

Example Usage

Here are some examples to illustrate the usage of the functions provided in pyspan:

import pandas as pd from pyspan import handle_nulls, remove_duplicates, remove_columns, auto_rename_columns, rename_dataframe_columns from pyspan import change_dt, detect_delimiter, split_column, impute, spell_check_dataframe from pyspan import detect_invalid_dates, detect_data_entry_errors, data_type_conversions, detect_outliers from pyspan import log_function_call, display_logs

Load a dataset

df = pd.read_csv('/content/GlobalSharkAttacks.csv')

Example usage of handle_nulls

df_filled = handle_nulls(df, columns=['Column1', 'Column2'], method='mean')

Example usage of remove_duplicates

df_unique = remove_duplicates(df)

Example usage of remove_columns

df_reduced = remove_columns(df, columns=['ColumnToRemove'])

Example usage of auto_rename_columns

auto_rename_columns(df)

Example usage of rename_dataframe_columns

rename_dict = {'OldName': 'NewName'} df_renamed_dict = rename_dataframe_columns(df, rename_dict)

Example usage of change_dt

df_formatted = change_dt(df, columns=['Date'], date_format='%d-%m-%Y', time_format='%I:%M %p')

Example usage of detect_delimiter

delimiter = detect_delimiter(df['ColumnWithDelimiters'])

Example usage of split_column

df_split = split_column(df, column_name='ColumnWithDelimiters')

Example usage of impute

df_imputed = impute(df, by='mean', columns=['Column1'])

Example usage of spell_check_dataframe

misspelled = spell_check_dataframe(df, dictionary='en_US', columns=['TextColumn'])

Example usage of detect_invalid_dates

invalid_dates = detect_invalid_dates(df['DateColumn'])

Example usage of detect_data_entry_errors

errors = detect_data_entry_errors(df, spellcheck_dict='en_US', date_columns=['DateColumn'], numeric_columns=['NumericColumn'])

Example usage of data_type_conversions

df_converted = data_type_conversions(df)

Example usage of detect_outliers

outliers = detect_outliers(df, method='iqr', threshold=1.5)

Example usage of display_logs

display_logs()

License

This package is licensed under the MIT License. See the LICENSE file for more details.

Contact

For issues or questions, please contact [[e-mail]].