Ak-dskit 1.0.6

A comprehensive data science toolkit with 221+ functions for ML workflows

🚀 Ak-dskit - A Unified Wrapper Library for Data Science & ML

Ak-dskit (import as dskit) is a comprehensive, community-driven, open-source Python library that wraps complex Data Science and ML operations into intuitive, user-friendly 1-line commands.

📝 Note: Install using pip install Ak-dskit, but import in Python as from dskit import dskit

Instead of writing hundreds of lines for cleaning, EDA, plotting, preprocessing, modeling, evaluation, and explainability, dskit makes everything simple, readable, reusable, and production-ready.

The goal is to bring a complete end-to-end Data Science ecosystem in one place with wrapper-style functions and classes, supporting everything from basic data manipulation to advanced AutoML.

---

🎯 Project Objective

To create a Python library that lets users perform complete Data Science workflows with minimal code:

from dskit import dskit

# Complete ML Pipeline in a few lines!
kit = dskit.load("data.csv")
kit.comprehensive_eda(target_col="target")  # EDA report
kit.clean()  # Clean data
kit.train_test_auto(target="target")  # Split data
kit.train_advanced("xgboost").auto_tune().evaluate().explain()  # Train, tune, evaluate, explain

The library remains:

• ✅ Simple: One-line commands for complex operations
• ✅ Comprehensive: 221 functions covering entire ML pipeline
• ✅ Extensible: Modular design for easy customization
• ✅ Beginner-friendly: Intuitive API with smart defaults
• ✅ Expert-ready: Advanced features and customization options
• ✅ Production-ready: Robust error handling and optimization

---

🎓 Learning Resources

📚 New to dskit? Start here!

1. Quick Start Guide - Get up and running in minutes
2. Demo Suite - 12 comprehensive demos covering all features
   • Start with 01_data_io_demo.py
   • Progress through 12_complete_pipeline_demo.py
3. API Reference - Detailed function documentation
4. Example Notebooks - Jupyter notebooks with real-world examples

🚀 Try It Now

# Install dskit
pip install Ak-dskit

# Run your first demo
cd demos
python 01_data_io_demo.py

---

📦 Installation

From PyPI (Recommended)

# Basic installation
pip install Ak-dskit

# Full installation with all optional dependencies
pip install Ak-dskit[full]

# Install specific feature sets
pip install Ak-dskit[visualization]  # Plotly support
pip install Ak-dskit[nlp]           # NLP utilities
pip install Ak-dskit[automl]        # AutoML algorithms

# Development installation
pip install Ak-dskit[dev]

From Source

git clone https://github.com/Programmers-Paradise/DsKit.git
cd DsKit
pip install -e .

Verify Installation

# Test the package
python test_package.py

# Check CLI
dskit --help

---

📦 Core Modules

dskit includes comprehensive modules for:

📁 Data I/O

• Multi-format loading (CSV, Excel, JSON, Parquet)
• Batch folder processing
• Smart data type detection

🧹 Data Cleaning

• Auto-detect and fix data types
• Smart missing value imputation
• Outlier detection and removal
• Column name standardization
• Text preprocessing and NLP utilities

📊 Exploratory Data Analysis

• Comprehensive EDA reports
• Data health scoring
• Interactive visualizations
• Statistical summaries
• Correlation analysis
• Missing data patterns

🔧 Feature Engineering

• Polynomial and interaction features
• Date/time feature extraction
• Binning and discretization
• Target encoding
• Dimensionality reduction (PCA)
• Text feature extraction
• Sentiment analysis

🤖 Machine Learning

• 15+ algorithms (including XGBoost, LightGBM, CatBoost)
• AutoML capabilities
• Hyperparameter optimization
• Cross-validation
• Ensemble methods
• Imbalanced data handling

📈 Visualization

• Static plots (matplotlib/seaborn)
• Interactive plots (plotly)
• Model performance charts
• Feature importance plots
• Advanced correlation heatmaps

🧠 Model Explainability

• SHAP integration
• Feature importance analysis
• Model performance metrics
• Error analysis
• Learning curves

📐 Hyperplane Analysis

• Algorithm-specific hyperplane visualization
• SVM margins and support vectors
• Logistic regression probability contours
• Perceptron misclassification highlighting
• LDA class centers and projections
• Linear regression residual analysis
• Multi-algorithm comparison tools

🎯 AutoML Features

• Automated preprocessing pipelines
• Model comparison and selection
• Hyperparameter tuning (Grid, Random, Bayesian, Optuna)
• Automated feature selection
• Pipeline optimization

---

🚀 Quick Start

Installation

# Basic installation
pip install Ak-dskit

# Full installation with all optional dependencies
pip install Ak-dskit[full]

# Development installation
git clone https://github.com/Programmers-Paradise/DsKit.git
cd DsKit
pip install -e .[dev,full]

✅ Verified Working Example

import pandas as pd
from dskit import dskit

# 1. Load data
kit = dskit.load("your_data.csv")

# 2. Basic data exploration
print(f"Data shape: {kit.df.shape}")
health_score = kit.data_health_check()
print(f"Data health score: {health_score}/100")

# 3. Data cleaning
kit = kit.fix_dtypes().fill_missing(strategy='auto').remove_outliers()

# 4. EDA (generates comprehensive report)
kit.comprehensive_eda(target_col="your_target_column")

# 5. Feature engineering
if 'date_column' in kit.df.columns:
    kit.create_date_features(['date_column'])
if 'text_column' in kit.df.columns:
    kit.advanced_text_clean(['text_column'])
    kit.sentiment_analysis(['text_column'])

# 6. Model training
X_train, X_test, y_train, y_test = kit.train_test_auto(target="your_target_column")
kit.train(model_name="random_forest")
kit.evaluate()

# 7. Model explainability
kit.explain()  # Generates SHAP explanations

Basic Usage

from dskit import dskit

# Load and explore data
kit = dskit.load("your_data.csv")
health_score = kit.data_health_check()  # Get data quality score
kit.comprehensive_eda(target_col="target")  # Full EDA report

# Clean and preprocess
kit.clean()  # Auto-clean: fix types, handle missing, normalize columns
# Create features manually
kit.create_polynomial_features(degree=2)
kit.create_date_features(["date_column"])

# Train and evaluate models
kit.train_test_auto(target="your_target")
kit.compare_models("your_target")  # Compare multiple algorithms
kit.train_advanced("xgboost").auto_tune()  # Train with hyperparameter tuning
kit.evaluate().explain()  # Evaluate and generate SHAP explanations

Advanced Features

# Advanced text processing
kit.sentiment_analysis(["text_column"])
kit.extract_text_features(["text_column"])
kit.generate_wordcloud("text_column")

# Feature engineering
kit.create_polynomial_features(degree=3)
kit.create_date_features(["date_column"])
kit.apply_pca(variance_threshold=0.95)

# AutoML
kit.auto_tune(method="optuna", max_evals=100)
best_models = kit.compare_models("target", task="classification")

# Advanced visualizations
kit.plot_feature_importance(top_n=20)
# Learning curves and validation curves are available through model validation module
# from dskit.model_validation import ModelValidator
# validator = ModelValidator()
# validator.learning_curve_analysis(model, X, y)

# Algorithm-specific hyperplane visualization
dskit.plot_svm_hyperplane(svm_model, X, y)  # SVM with margins
dskit.plot_logistic_hyperplane(lr_model, X, y)  # Probability contours
dskit.plot_perceptron_hyperplane(perceptron_model, X, y)  # Misclassified points

# Compare multiple algorithm hyperplanes
models = {'SVM': svm, 'LR': lr, 'Perceptron': perceptron}
dskit.compare_algorithm_hyperplanes(models, X, y)

---

📚 Complete Feature Documentation

🧩 IMPLEMENTED FEATURES (All Tasks Complete)

Each task below is numbered and written in simple language with enough theory so that any contributor — even new ones — can understand exactly what to build.

---

🎯 Demos & Examples

📁 Comprehensive Demo Suite

We've created 12 detailed demo files showcasing every major feature of dskit! Each demo is self-contained, well-documented, and includes practical examples.

👉 Explore the Demos Folder

Core Functionality Demos (1-6)

Demo	Description	Key Features
01_data_io_demo.py	Data I/O Operations	Load/save CSV, Excel, JSON, Parquet, batch loading
02_data_cleaning_demo.py	Data Cleaning	Type fixing, missing values, outliers, text cleaning
03_eda_demo.py	Exploratory Analysis	Statistics, health checks, comprehensive EDA
04_visualization_demo.py	Visualizations	Histograms, boxplots, correlations, pairplots
05_preprocessing_demo.py	Preprocessing	Encoding, scaling, train-test split
06_modeling_demo.py	ML Modeling	Training, comparison, tuning, evaluation

Advanced Feature Demos (7-12)

Demo	Description	Key Features
07_feature_engineering_demo.py	Feature Engineering	Polynomial, date features, PCA, target encoding
08_nlp_demo.py	NLP Utilities	Text stats, cleaning, features, sentiment
09_advanced_visualization_demo.py	Advanced Plots	Feature importance, correlations, patterns
10_automl_demo.py	AutoML	Random/Grid/Bayesian search, optimization
11_hyperplane_demo.py	Hyperplanes	SVM, logistic regression visualization
12_complete_pipeline_demo.py	End-to-End	Complete ML workflow from data to deployment

Quick Start with Demos

# Navigate to demos folder
cd demos

# Run a specific demo
python 01_data_io_demo.py

# Run all demos interactively
python run_all_demos.py

📖 Full Demo Documentation

---

📖 Examples & Tutorials

Complete ML Pipeline Example

import pandas as pd
from dskit import dskit

# 1. Load and explore
kit = dskit.load("customer_data.csv")
health_score = kit.data_health_check()  # Returns: 85.3/100

# 2. Comprehensive EDA
kit.comprehensive_eda(target_col="churn", sample_size=1000)
kit.generate_profile_report("eda_report.html")  # Automated EDA report

# 3. Advanced text processing (if text columns exist)
kit.advanced_text_clean(["feedback"])
kit.sentiment_analysis(["feedback"])
kit.extract_text_features(["feedback"])

# 4. Feature engineering
kit.create_date_features(["registration_date"])
kit.create_polynomial_features(degree=2, interaction_only=True)
kit.create_binning_features(["age", "income"], n_bins=5)

# 5. Preprocessing
kit.clean()  # Auto-clean pipeline
# Handle imbalanced data if needed
# from dskit.advanced_modeling import handle_imbalanced_data
# X_balanced, y_balanced = handle_imbalanced_data(X, y, method="smote")

# 6. Model training and optimization
X_train, X_test, y_train, y_test = kit.train_test_auto("churn")
comparison = kit.compare_models("churn")  # Compare 10+ algorithms
kit.train_advanced("xgboost").auto_tune(method="optuna", max_evals=50)

# 7. Evaluation and explainability
kit.evaluate().explain()  # Comprehensive evaluation + SHAP
kit.plot_feature_importance()
kit.cross_validate(cv=5)

NLP Pipeline Example

# Text analysis workflow
kit = dskit.load("reviews.csv")
kit.text_stats(["review_text"])  # Basic text statistics
kit.advanced_text_clean(["review_text"], remove_urls=True, expand_contractions=True)
kit.sentiment_analysis(["review_text"])  # Add sentiment scores
kit.generate_wordcloud("review_text", max_words=100)
kit.extract_keywords("review_text", top_n=20)

Time Series Feature Engineering

# Date/time feature extraction
kit.create_date_features(["transaction_date"])
# Creates: year, month, day, weekday, quarter, is_weekend columns

kit.create_aggregation_features("customer_id", ["amount"], ["mean", "std", "count"])
# Creates aggregated features grouped by customer

---

🎯 AutoML Capabilities

dskit includes comprehensive AutoML features:

• Automated Preprocessing: Smart data cleaning and feature engineering
• Model Selection: Automatic algorithm comparison and selection
• Hyperparameter Optimization: Grid, Random, Bayesian, and Optuna-based tuning
• Feature Selection: Univariate, RFE, and embedded methods
• Ensemble Methods: Voting classifiers and advanced ensembles
• Performance Optimization: Cross-validation and learning curve analysis

---

📊 Supported Algorithms

Classification & Regression

• Traditional: Random Forest, Gradient Boosting, SVM, KNN, Naive Bayes
• Advanced: XGBoost, LightGBM, CatBoost, Neural Networks
• Ensemble: Voting Classifiers, Stacking, Bagging

Preprocessing

• Scaling: Standard, MinMax, Robust, Quantile
• Encoding: Label, One-Hot, Target, Binary
• Imputation: Mean, Median, Mode, KNN, Iterative
• Feature Selection: SelectKBest, RFE, RFECV, Embedded

---

🔧 Configuration

dskit supports flexible configuration:

# Global configuration
from dskit.config import set_config
set_config({
    'visualization_backend': 'plotly',  # or 'matplotlib'
    'auto_save_plots': True,
    'default_test_size': 0.2,
    'random_state': 42,
    'n_jobs': -1
})

# Method-specific parameters
kit.auto_tune(method="optuna", max_evals=100, timeout=3600)
kit.comprehensive_eda(sample_size=5000, include_correlations=True)

---

🔧 Troubleshooting Common Issues

Import Errors

# ❌ This might fail with import errors
from dskit import non_existent_function

# ✅ Import correctly
from dskit import dskit, load, fix_dtypes, quick_eda

Method Chaining

# ❌ Some methods don't return self
result = kit.missing_summary().fill_missing()  # Error!

# ✅ Correct approach
missing_info = kit.missing_summary()  # Returns DataFrame
kit = kit.fill_missing()  # Returns dskit object

Data Loading

# ❌ File not found
kit = dskit.load("non_existent_file.csv")

# ✅ Check file exists first
import os
if os.path.exists("data.csv"):
    kit = dskit.load("data.csv")
else:
    print("File not found!")

Target Column Issues

# ❌ Target column doesn't exist
kit.train_test_auto(target="non_existent_column")

# ✅ Check columns first
print("Available columns:", kit.df.columns.tolist())
if "target" in kit.df.columns:
    X_train, X_test, y_train, y_test = kit.train_test_auto(target="target")

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

📚 Complete Documentation

For comprehensive guides, implementation details, and technical documentation, visit our organized documentation center:

📁 docs/ - Complete Documentation Center

Key Documentation Highlights

• Feature Engineering Implementation Guide - Deep dive into how dskit creates features and backend implementation
• Complete ML Pipeline Comparison - Traditional vs Dskit approaches with 61% code reduction analysis
• API Reference - Complete API documentation
• Quick Test Summary - Getting started guide

Notebooks & Examples

• complete_ml_dskit.ipynb - Complete ML pipeline using dskit
• complete_ml_traditional.ipynb - Traditional ML pipeline for comparison
• dskit_vs_traditional_comparison.ipynb - Side-by-side comparison

---

Acknowledgments

• Built on top of excellent libraries: pandas, scikit-learn, matplotlib, seaborn, plotly
• Inspired by the need for simplified data science workflows
• Community-driven development with contributions from data scientists worldwide

---

Ak-dskit (dskit) - Making Data Science Simple, Comprehensive, and Accessible! 🚀