dataanalysts 0.2.4

An advanced Python data analysis library with enhanced cleaning, transformation, and visualization.

DataAnalysts Package

DataAnalysts is a robust and versatile Python library meticulously designed to simplify and enhance the data analysis process. It caters to users across diverse domains, including students, professional data analysts, researchers, and enthusiasts. The library integrates powerful modules for data cleaning, transformation, and visualization, enabling seamless handling of datasets with minimal coding effort.

Whether you're dealing with messy datasets or preparing sophisticated visualizations, DataAnalysts offers an intuitive and interactive interface to perform your tasks with precision and efficiency.

🚀 Key Features

Data Cleaning:

• Handle Missing Values:

  • Supports mean, median, and mode strategies for numeric columns.
  • Automatically fills missing categorical data using mode.

• Remove Duplicates:

  • Eliminates duplicate rows to ensure data integrity.

• Fill Unknown Values:

  • Replaces missing categorical data with 'Unknown' and numeric data with 0.

• Convert Strings to Numeric:

  • Converts applicable string columns to numeric where possible.

• Impute Missing Values by Group:

  • Fills missing values within groups defined by another column (e.g., by city or category).

• Drop Low Variance Columns:

  • Removes columns with variance below a user-specified threshold.

• Handle Outliers:

  • Detects and caps outliers in numeric columns using the IQR method.

• Standardize Column Names:

  • Renames columns to lowercase, replaces spaces with underscores, and removes extra spaces.

• Encode Categorical Variables:

  • Converts categorical columns to numeric codes.

• Feature Scaling (Normalization):

  • Scales numeric columns to have a mean of 0 and a standard deviation of 1.

---

Interactive Cleaning:

• Customizable Options:

  • Perform cleaning tasks step-by-step through a user-friendly menu interface.

• Menu Options:

  • Handle missing values, remove duplicates, drop columns, rename columns, handle outliers, encode categorical variables, fill unknown values, convert strings to numeric, impute missing values by group, drop low variance columns, and scale features.

---

Data Transformation:

• Scaling:

  • Standard, Min-Max, and Robust scaling strategies for numeric columns.

• Encoding:

  • Label encoding for categorical columns.

• Dimensionality Reduction:

  • Principal Component Analysis (PCA) to reduce dataset dimensions.

• Duplicate Removal:

  • Automatically remove duplicate rows.

• Low-Variance Feature Removal:

  • Remove features with variance below a defined threshold.

• Interactive Transformation:

  • Choose transformation steps interactively.

---

Data Visualization:

• Histogram:

  • Plot a histogram with advanced customization for bins, labels, and title.

• Bar Chart:

  • Generate bar charts with customizable sizes, labels, and colors.

• Line Plot:

  • Create line plots with options for markers, colors, and labels.

• Scatter Plot:

  • Generate scatter plots with hue-based grouping for better insights.

• Heatmap:

  • Visualize correlation matrices using customizable heatmaps.

• Pair Plot:

  • Plot pairwise relationships between numeric columns.

• Box Plot:

  • Create box plots to visualize data distribution and outliers.

• Violin Plot:

  • Generate violin plots to show data distribution with additional density insights.

---

Interactive Visualization:

• Customizable Options:

  • Perform visualizations interactively through a user-friendly menu interface.

• Menu Options:

  • Choose from histograms, bar charts, line plots, scatter plots, heatmaps, pair plots, box plots, and violin plots.

---

Data Loading:

• CSV Files:

  • Easily load datasets from CSV files with automatic logging.

• Excel Files:

  • Load data from Excel sheets with customizable sheet selection.

---

Error Handling:

• Robust Exception Handling:

  • Provides clear error messages for debugging and ensures smooth execution.

---

Interactive Tools:

• Data Cleaning:

  • Step-by-step interactive data cleaning options.

• Data Transformation:

  • Hands-on transformation with flexible menu options.

• Data Visualization:

  • Interactive plotting with multiple customization options.

---

🛠️ Installation Steps

1. Install the Package from PyPI

To use the library in Google Colab or your local environment, install it directly from PyPI:

pip install dataanalysts
!pip install dataanalysts

---

💡 Usage Examples

1. Import the Library

import dataanalysts as da
import pandas as pd

2. Load Data

df = da.csv('data.csv')
df_excel = da.excel('data.xlsx', sheet_name='Sheet1')

3. Data Cleaning

This module allows users to handle missing values, remove duplicates, encode categorical variables, handle outliers, and perform other essential data cleaning operations with specific strategies or through an interactive menu.

---

Key Features

1. Remove Duplicates: Removes duplicate rows from the dataset.
2. Fill Unknown Values: Fills missing values in categorical and numeric columns with specified defaults.
3. Handle Missing Values: Fills missing numeric values using mean, median, or mode strategies.
4. Convert to Numeric: Converts applicable string columns to numeric where possible.
5. Impute by Group: Imputes missing values based on groups defined by another column.
6. Drop Low Variance Columns: Drops columns with variance below a specified threshold.
7. Handle Outliers: Caps outliers in numeric columns using the IQR method.
8. Interactive Cleaning: Allows users to clean datasets step-by-step through a user-friendly menu.

---

Syntax and Examples

1. Remove Duplicates

Removes duplicate rows to ensure data integrity.

Syntax:

from dataanalysts.cleaner import clean

# Remove duplicate rows
df_cleaned = clean(df, strategy='remove_duplicates')

Example:

import pandas as pd
from dataanalysts.cleaner import clean

data = {
    "Name": ["Alice", "Bob", "Alice"],
    "Age": [25, 30, 25],
    "City": ["NY", "LA", "NY"]
}
df = pd.DataFrame(data)
df_cleaned = clean(df, strategy='remove_duplicates')
print(df_cleaned)

---

2. Fill Unknown Values

Fills missing categorical values with 'Unknown' and numeric values with 0.

Syntax:

# Fill unknown values
df_cleaned = clean(df, strategy='fill_unknown')

Example:

import pandas as pd
from dataanalysts.cleaner import clean

data = {
    "Name": ["Alice", None, "Charlie"],
    "Age": [25, None, 35]
}
df = pd.DataFrame(data)
df_cleaned = clean(df, strategy='fill_unknown')
print(df_cleaned)

---

3. Handle Missing Values

Fills missing numeric values using mean, median, or mode strategies.

Syntax:

# Fill missing values with mean
df_cleaned = clean(df, strategy='mean')

# Fill missing values with median
df_cleaned = clean(df, strategy='median')

# Fill missing values with mode
df_cleaned = clean(df, strategy='mode')

Example:

import pandas as pd
from dataanalysts.cleaner import clean

data = {
    "Age": [25, None, 35],
    "Salary": [50000, None, 70000]
}
df = pd.DataFrame(data)
df_cleaned = clean(df, strategy='mean')
print(df_cleaned)

---

4. Convert to Numeric

Converts applicable string columns to numeric values.

Syntax:

# Convert string columns to numeric
df_cleaned = clean(df, strategy='convert_to_numeric')

Example:

import pandas as pd
from dataanalysts.cleaner import clean

data = {
    "Name": ["Alice", "Bob"],
    "Age": ["25", "30"],
    "Salary": ["50000", "60000"]
}
df = pd.DataFrame(data)
df_cleaned = clean(df, strategy='convert_to_numeric')
print(df_cleaned)

---

5. Impute Missing Values by Group

Fills missing values within groups defined by another column.

Syntax:

# Impute missing values by group
df_cleaned = clean(df, strategy='impute_by_group', group_column='City')

Example:

import pandas as pd
from dataanalysts.cleaner import clean

data = {
    "City": ["NY", "LA", "NY", "Chicago"],
    "Age": [25, 30, None, 35]
}
df = pd.DataFrame(data)
df_cleaned = clean(df, strategy='impute_by_group', group_column='City')
print(df_cleaned)

---

6. Drop Low Variance Columns

Removes columns with variance below a specified threshold.

Syntax:

# Drop columns with low variance
df_cleaned = clean(df, strategy='drop_low_variance', threshold=0.01)

Example:

import pandas as pd
from dataanalysts.cleaner import clean

data = {
    "Constant": [1, 1, 1],
    "Variable": [1, 2, 3]
}
df = pd.DataFrame(data)
df_cleaned = clean(df, strategy='drop_low_variance', threshold=0.01)
print(df_cleaned)

---

7. Handle Outliers

Caps outliers in numeric columns using the IQR method.

Syntax:

# Handle outliers
df_cleaned = clean(df, strategy='handle_outliers')

Example:

import pandas as pd
from dataanalysts.cleaner import clean

data = {
    "Age": [25, 30, 35, 120],
    "Salary": [50000, 60000, 70000, 1000000]
}
df = pd.DataFrame(data)
df_cleaned = clean(df, strategy='handle_outliers')
print(df_cleaned)

---

8. Interactive Cleaning

Provides an interactive menu for step-by-step cleaning.

Syntax:

from dataanalysts.cleaner import interactive_clean

# Perform interactive cleaning
df_cleaned = interactive_clean(df)

Example:

import pandas as pd
from dataanalysts.cleaner import interactive_clean

data = {
    "Name": ["Alice", None, "Charlie"],
    "Age": [25, None, 35]
}
df = pd.DataFrame(data)
df_cleaned = interactive_clean(df)

---

Logging

• All operations are logged to the cleaner.log file.
• Errors and cleaning steps are logged for tracking and debugging purposes.

This module simplifies the process of cleaning datasets, ensuring consistency and accuracy in preprocessing tasks.

4. Data Transformation

The Data Transformation Module enables comprehensive data preprocessing and transformation for datasets, including scaling, dimensionality reduction, encoding, and more. The module supports both direct and interactive transformation methods.

---

Key Features

• Scaling: Standard, Min-Max, and Robust scaling strategies for numeric columns.
• Encoding: Label encoding for categorical columns.
• Dimensionality Reduction: Principal Component Analysis (PCA) to reduce dataset dimensions.
• Duplicate Removal: Automatically remove duplicate rows.
• Low-Variance Feature Removal: Remove features with variance below a defined threshold.
• Interactive Transformation: Choose transformation steps interactively.

---

Syntax and Examples

1. Scaling

Scales numeric columns based on the selected strategy:

• Standard Scaling: Centers data around mean (0) with standard deviation (1).
• Min-Max Scaling: Scales data to a range of [0, 1].
• Robust Scaling: Handles outliers by scaling data based on the interquartile range (IQR).

Syntax:

import dataanalysts as da

# Standard Scaling
df_transformed = da.transform(df, strategy='standard')

# Min-Max Scaling
df_transformed = da.transform(df, strategy='minmax')

# Robust Scaling
df_transformed = da.transform(df, strategy='robust')

---

2. Encoding

Encodes categorical columns into numeric values using label encoding. This is particularly useful for machine learning models that require numeric data.

Syntax:

# Encode categorical columns
df_transformed = da.transform(df, encode_categorical=True)

---

3. Duplicate Removal

Automatically removes duplicate rows from the dataset.

Syntax:

# Remove duplicate rows
df_transformed = da.transform(df, remove_duplicates=True)

---

4. Low-Variance Feature Removal

Removes features with variance below a specified threshold to reduce noise in the data.

Syntax:

# Remove features with variance below 0.01
df_transformed = da.transform(df, remove_low_variance=True, variance_threshold=0.01)

---

5. Dimensionality Reduction (PCA)

Uses Principal Component Analysis to reduce the number of features while retaining most of the variance in the dataset.

Syntax:

# Apply PCA to retain 3 components
df_pca = da.transform(df_transformed, reduce_dimensionality=True, n_components=3)

---

6. Interactive Transformation

Provides an interactive menu for selecting transformation steps one at a time.

Menu Options:

1. Apply Standard Scaling
2. Apply Min-Max Scaling
3. Apply Robust Scaling
4. Encode Categorical Columns
5. Remove Duplicates
6. Remove Low-Variance Features
7. Apply PCA for Dimensionality Reduction
8. Exit Transformation

Syntax:

# Perform interactive transformation
df_interactive_transform = da.interactive_transform(df)

---

Comprehensive Example

Here’s an end-to-end example combining multiple transformations:

import dataanalysts as da
import pandas as pd

# Sample dataset
data = {
    'Age': [25, 30, 35, 40, 45],
    'Salary': [50000, 60000, 70000, 80000, 90000],
    'Department': ['HR', 'IT', 'Finance', 'IT', 'HR']
}
df = pd.DataFrame(data)

# Step 1: Apply standard scaling
df_transformed = da.transform(df, strategy='standard')

# Step 2: Apply PCA to reduce dimensions to 2 components
df_pca = da.transform(df_transformed, reduce_dimensionality=True, n_components=2)

# Step 3: Perform additional transformations interactively
df_final = da.interactive_transform(df_pca)

print(df_final)

---

Logging

• Logs are stored in the transformer.log file.
• Each transformation step is logged with details about the operation and parameters used.
• Errors during transformations are also logged for debugging purposes.

5. Data Visualization

Data Visualization

The Data Visualization Module provides advanced tools for creating insightful and customized visual representations of your dataset. With this module, you can generate a variety of plots, including histograms, scatter plots, heatmaps, and more, with customization options for size, titles, and styles.

---

Key Features

• Histogram: Visualize the distribution of a single numeric column.
• Bar Chart: Compare values across categories.
• Line Chart: Display trends over time or sequential data.
• Scatter Plot: Show relationships between two numeric columns.
• Heatmap: Visualize correlations between numeric columns.
• Pair Plot: Display pairwise relationships in a dataset.
• Box Plot: Compare distributions of a numeric column across categories.
• Violin Plot: Combine box plot and density plot for richer insights.
• Interactive Visualization: Select and generate plots interactively.

---

Syntax and Examples

1. Histogram

Visualize the distribution of a single numeric column.

Syntax:

da.histogram(df, column='age', bins=30, kde=True)

Customization Options:

• bins: Number of bins for the histogram.
• kde: Whether to display the Kernel Density Estimate.
• size: Tuple specifying figure size.
• title_fontsize: Font size for the title.
• axis_fontsize: Font size for axis labels.
• custom_title: Custom title for the chart.

---

2. Bar Chart

Compare values across categories.

Syntax:

da.barchart(df, x_col='city', y_col='population')

Customization Options:

• size: Tuple specifying figure size.
• title_fontsize: Font size for the title.
• axis_fontsize: Font size for axis labels.
• custom_title: Custom title for the chart.

---

3. Line Chart

Display trends over time or sequential data.

Syntax:

da.linechart(df, x_col='date', y_col='sales')

Customization Options:

• size: Tuple specifying figure size.
• title_fontsize: Font size for the title.
• axis_fontsize: Font size for axis labels.
• custom_title: Custom title for the chart.

---

4. Scatter Plot

Show relationships between two numeric columns.

Syntax:

da.scatter(df, x_col='height', y_col='weight', hue='gender')

Customization Options:

• hue: Column for color encoding.
• size: Tuple specifying figure size.
• title_fontsize: Font size for the title.
• axis_fontsize: Font size for axis labels.
• custom_title: Custom title for the chart.

---

5. Heatmap

Visualize correlations between numeric columns.

Syntax:

da.heatmap(df)

Customization Options:

• annot: Whether to annotate the heatmap with correlation values.
• cmap: Colormap for the heatmap.
• size: Tuple specifying figure size.
• title_fontsize: Font size for the title.
• custom_title: Custom title for the chart.

---

6. Pair Plot

Display pairwise relationships in a dataset.

Syntax:

da.pairplot(df, hue='category')

Customization Options:

• hue: Column for color encoding.
• size: Tuple specifying figure size for each subplot.
• title_fontsize: Font size for the title.
• custom_title: Custom title for the chart.

---

7. Box Plot

Compare distributions of a numeric column across categories.

Syntax:

da.boxplot(df, x_col='region', y_col='sales')

Customization Options:

• size: Tuple specifying figure size.
• title_fontsize: Font size for the title.
• axis_fontsize: Font size for axis labels.
• custom_title: Custom title for the chart.

---

8. Violin Plot

Combine box plot and density plot for richer insights.

Syntax:

da.violinplot(df, x_col='region', y_col='sales')

Customization Options:

• size: Tuple specifying figure size.
• title_fontsize: Font size for the title.
• axis_fontsize: Font size for axis labels.
• custom_title: Custom title for the chart.

---

9. Interactive Visualization

Provides an interactive menu for generating various plots one at a time.

Menu Options:

1. Histogram
2. Bar Chart
3. Line Plot
4. Scatter Plot
5. Heatmap
6. Pair Plot
7. Box Plot
8. Violin Plot
9. Exit Visualization

Syntax:

# Perform interactive visualization
da.interactive_plot(df)

---

Comprehensive Example

Here’s how you can use the visualizer functions to create multiple plots:

import dataanalysts as da
import pandas as pd

# Sample dataset
data = {
    'age': [25, 30, 35, 40, 45],
    'salary': [50000, 60000, 70000, 80000, 90000],
    'city': ['NY', 'LA', 'SF', 'CHI', 'HOU'],
    'gender': ['M', 'F', 'F', 'M', 'M']
}
df = pd.DataFrame(data)

# Histogram
da.histogram(df, column='age', bins=20, kde=True)

# Bar Chart
da.barchart(df, x_col='city', y_col='salary')

# Scatter Plot
da.scatter(df, x_col='age', y_col='salary', hue='gender')

# Heatmap
da.heatmap(df)

# Interactive Visualization
da.interactive_plot(df)

---

Logging

• Logs are stored in the visualizer.log file.
• Each visualization step is logged with details about the operation and parameters used.
• Errors during visualizations are also logged for debugging purposes.

---

This module provides highly customizable and interactive visualizations to gain insights from your data effectively.

---

🤝 Contributing

Contributions are welcome! Please submit a pull request via our GitHub Repository.

---

📜 License

This project is licensed under the MIT License. See the LICENSE file for details.

---

🛠️ Support

If you encounter any issues, feel free to open an issue on our GitHub Issues page.