smart-datalyzer 0.1.1

Data analysis and reporting toolkit

Smart Datalyzer

Smart Datalyzer is an intelligent, automated toolkit for comprehensive data analysis, visualization, and reporting. It provides ML readiness scoring, advanced statistical diagnostics, and publication-quality visualizations with minimal effort.

🚀 Key Features

📊 Data Quality & Profiling

• Smart Dataset Loading: Automatic detection of CSV/XLSX files with type inference
• Duplicate Detection: Identify and report duplicate rows
• Mixed Type Detection: Find columns with inconsistent data types
• Auto Type Conversion: Intelligent conversion of string columns to numeric
• Missing Value Analysis: Detection and imputation suggestions
• Constant Column Detection: Identify features with zero variance
• Scaling Issue Detection: Flag features with extreme value ranges

🎯 Target-Aware Analysis (Multiple Targets Support)

• Target Leakage Detection: Identify features that leak target information (>95% accuracy)
• Class Imbalance Analysis: Compute imbalance ratios and distribution statistics
• Feature-Target Association: Statistical tests (ANOVA, Kruskal-Wallis, Chi-square)
• Sensitivity Analysis: Permutation importance for feature ranking
• Model Suggestion: Automatic recommendation (Regression vs Classification)

📈 Statistical Diagnostics

• Normality Testing: Shapiro-Wilk, D'Agostino, Kolmogorov-Smirnov tests with QQ plots
• Outlier Detection: Z-score based detection with percentage reporting
• Correlation Analysis: Pearson, Spearman, Kendall correlation matrices
• VIF Computation: Variance Inflation Factor for multicollinearity detection
• Mutual Information: Feature importance via mutual information scores
• Covariance Matrix: Full covariance analysis with CSV export
• High Correlation Flagging: Automatic detection of correlated pairs (>0.9)

📉 Visualization Suite

• Distribution Plots: Histograms with KDE overlays
• Box Plots: Outlier visualization with quartile analysis
• Violin Plots: Distribution density visualization
• Swarm Plots: Individual data point overlay on boxplots
• QQ Plots: Quantile-quantile plots for normality assessment
• Correlation Heatmaps: Multiple correlation methods with annotations
• Feature Importance Charts: RandomForest-based importance ranking
• PCA Variance Plots: Principal component analysis visualization
• t-SNE Scatter Plots: 2D dimensionality reduction visualization

📝 Reporting & Export

• Interactive HTML Reports: Comprehensive analysis with embedded visualizations
• JSON Export: Machine-readable summary statistics
• PDF Generation: Publication-ready reports (optional)
• Plot Export: High-resolution PNG plots (300 DPI)
• Caching System: Smart caching for faster re-analysis

🤖 Smart Auto Mode

• Automatic feature engineering recommendations
• ML readiness scoring (0-100)
• Actionable improvement suggestions
• Complete pipeline execution with single flag

📦 Installation

From Source (Recommended)

# Clone the repository
git clone https://github.com/mehmoodulhaq570/smart-datalyzer.git
cd smart-datalyzer

# Install build tools
pip install build

# Build the package
python -m build

# Install
pip install dist/smart_datalyzer-0.1.1-py3-none-any.whl

Development Install

pip install -e .

🎮 Usage

Basic Usage (Single Target)

python -m smart-datalyzer data.xlsx "target_column"

Or using the installed command:

smart-datalyzer data.xlsx "target_column"

Multiple Target Columns

python -m smart-datalyzer data.csv "target1" "target2" "target3"

Command Line Arguments

python -m smart-datalyzer <file> <target> [OPTIONS]
# or
smart-datalyzer <file> <target> [OPTIONS]

Arguments:
  file                    Path to dataset (CSV or XLSX)
  target                  Target column name(s) - space separated for multiple

Options:
  --stats                 Run detailed statistical analysis
  --outliers             Detect and report outliers
  --leakage              Detect target leakage features
  --imbalance            Check class imbalance
  --plots                Generate all visualization plots
  --report               Generate interactive HTML/JSON report
  --auto                 Run full automatic analysis (recommended)
  --max_rows N           Limit rows to read (default: 100000)
  --output_dir DIR       Output directory (default: "reports")

Examples

Quick Analysis:

python -m smart-datalyzer sales.xlsx "Revenue" --auto

Detailed Statistical Report:

smart-datalyzer customers.csv "Churn" --stats --plots --report

Multiple Targets with Custom Output:

smart-datalyzer experiment.xlsx "Outcome1" "Outcome2" --auto --output_dir results

Outlier & Leakage Detection:

python -m smart-datalyzer medical.csv "Disease" --outliers --leakage

📊 Output Structure

reports/
├── plots/
│   ├── *_distribution.png      # Distribution histograms
│   ├── *_boxplot.png           # Box plots
│   ├── *_violinplot.png        # Violin plots
│   ├── *_swarmplot.png         # Swarm plots
│   ├── *_qqplot.png            # QQ plots
│   ├── correlation_*.png       # Correlation heatmaps
│   ├── feature_importance.png  # Feature importance chart
│   ├── pca_variance.png        # PCA analysis
│   └── tsne_scatter.png        # t-SNE visualization
├── report.html                 # Interactive HTML report
├── summary.json                # JSON summary statistics
├── covariance_matrix.csv       # Covariance matrix
└── .cache/                     # Analysis cache

🧰 Python API Usage

from datalyzer.utils import load_dataset
from datalyzer.stats import feature_statistics, detect_outliers
from datalyzer.plots import plot_distributions, plot_correlation

# Load data
df = load_dataset("data.csv")

# Get statistics
stats, readiness, suggestions = feature_statistics(df)
print(f"ML Readiness Score: {readiness}/100")

# Detect outliers
outliers = detect_outliers(df, df.select_dtypes(include=['float64', 'int64']).columns)

# Generate plots
plot_paths = plot_distributions(df, plots_dir="reports/plots")
correlation_paths = plot_correlation(df, plots_dir="reports/plots")

🔧 Dependencies

Core Requirements

• pandas - Data manipulation
• numpy - Numerical computing
• scipy - Statistical functions
• statsmodels - Advanced statistics
• scikit-learn - Machine learning utilities
• matplotlib - Plotting backend
• seaborn - Statistical visualizations
• rich - Terminal formatting

See requirements.txt for complete list.

🎨 Features in Detail

ML Readiness Score

Smart Datalyzer computes an ML readiness score (0-100) based on:

• Missing value percentage
• Constant features
• Numeric vs categorical balance
• Duplicate rows
• Data quality issues

Caching System

Automatically caches analysis results using SHA256 hashing for:

• Faster re-analysis of same datasets
• Incremental updates
• Reduced computation time

Smart Type Inference

Automatically detects and suggests:

• Numeric columns stored as strings
• Categorical features with high cardinality
• Date/time columns
• Mixed-type columns

👨‍💻 Author

Mehmood Ul Haq
Email: [e-mail]
GitHub: @mehmoodulhaq570

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please read CODE_OF_CONDUCT.md first.

🔒 Security

For security issues, please see SECURITY.md.

📝 Changelog

v0.1.1 (Current)

• Fixed swarm plot performance issues with large datasets (added sampling limit of 2000 points)
• Fixed filename sanitization for plots with special characters
• Improved visualization generation speed
• Skip class imbalance check for targets with >10 unique values

v0.1.0

• Initial release
• Multiple target column support
• Comprehensive statistical analysis
• Advanced visualization suite
• Smart auto-analysis mode
• Caching system
• Interactive HTML reports

🙏 Acknowledgments

Built with modern Python data science stack and best practices for automated data analysis.