clustertk 0.5.0

A comprehensive toolkit for cluster analysis with full pipeline support

ClusterTK

A comprehensive toolkit for cluster analysis with full pipeline support

ClusterTK is a Python library designed to streamline the entire cluster analysis workflow. It provides a unified, easy-to-use interface for data preprocessing, feature selection, dimensionality reduction, clustering, evaluation, and interpretation.

📋 Quick Navigation

• Quick Start ⭐ - Get started in 30 seconds
• Installation - Install options
• Visualization - Plot examples
• Pipeline Components - Detailed component reference
• Examples - Jupyter notebooks

📚 Table of Contents (click to expand)

• Features
• Installation
  • Basic Installation
  • With Visualization Support
  • Full Installation
  • Development Installation
• Quick Start
• Step-by-Step Usage
• Pipeline Components
  • 1. Preprocessing
  • 2. Feature Selection
  • 3. Dimensionality Reduction
  • 4. Clustering
  • 5. Evaluation
  • 6. Interpretation
• Visualization
• Export Results
• Advanced Usage
• Contributing
• License

Features

• 🔄 Complete Pipeline: One-line solution from raw data to cluster insights
• 🛠️ Modular Design: Use individual components or the full pipeline
• 📊 Multiple Algorithms: K-Means, GMM, Hierarchical, DBSCAN
• 🎯 Automatic Optimization: Auto-selection of optimal cluster numbers
• 📈 Rich Evaluation: Comprehensive metrics (Silhouette, Calinski-Harabasz, Davies-Bouldin)
• 🎨 Optional Visualization: Beautiful plots without mandatory heavy dependencies
• 🔍 Cluster Interpretation: Automatic profiling and naming suggestions
• 📁 Export & Reports: CSV, JSON exports, HTML reports with embedded plots
• 💾 Save/Load: Serialize and reload fitted pipelines

Installation

Basic Installation (Core functionality)

pip install clustertk

With Visualization Support

pip install clustertk[viz]

Full Installation (All features)

pip install clustertk[all]

Development Installation

git clone https://github.com/alexeiveselov92/clustertk.git
cd clustertk
pip install -e .[dev]

Quick Start

import pandas as pd
from clustertk import ClusterAnalysisPipeline

# Load your data
df = pd.read_csv('your_data.csv')

# Create and configure pipeline
pipeline = ClusterAnalysisPipeline(
    handle_missing='median',          # Handle missing values
    correlation_threshold=0.85,       # Remove highly correlated features
    pca_variance=0.9,                 # Keep 90% of variance
    clustering_algorithm='kmeans',    # Use K-Means
    n_clusters=None,                  # Auto-detect optimal number
    verbose=True
)

# Run complete analysis
pipeline.fit(df, feature_columns=['col1', 'col2', 'col3'])

# Get results
labels = pipeline.labels_                    # Cluster assignments
profiles = pipeline.cluster_profiles_        # Cluster profiles
metrics = pipeline.metrics_                  # Quality metrics

# Export results
pipeline.export_results('results.csv')       # CSV with data + labels
pipeline.export_results('results.json', format='json')  # JSON with metadata
pipeline.export_report('report.html')        # HTML report with plots

# Visualize (if viz dependencies installed)
# Note: In Jupyter, use display() for multiple plots in one cell
from IPython.display import display
display(pipeline.plot_clusters_2d())
display(pipeline.plot_cluster_heatmap())  # or plot_cluster_radar()

Step-by-Step Usage

You can also run the pipeline step-by-step for more control:

pipeline = ClusterAnalysisPipeline()

# Step 1: Preprocess data
pipeline.preprocess(df, feature_columns=['col1', 'col2', 'col3'])

# Step 2: Select features
pipeline.select_features()

# Step 3: Reduce dimensions
pipeline.reduce_dimensions()

# Step 4: Find optimal number of clusters
pipeline.find_optimal_clusters()

# Step 5: Perform clustering
pipeline.cluster(n_clusters=5)

# Step 6: Create cluster profiles
pipeline.create_profiles(category_mapping={
    'behavioral': ['sessions', 'duration'],
    'engagement': ['clicks', 'likes']
})

# Access intermediate results
preprocessed_data = pipeline.data_preprocessed_
pca_components = pipeline.data_reduced_

Pipeline Components

🔍 Click to see detailed component documentation

1. Preprocessing

• Missing Values: Median, mean, drop, or custom imputation
• Outliers: IQR detection, robust scaling, clipping, or removal
• Scaling: StandardScaler, RobustScaler, MinMaxScaler, or auto-selection
• Transformations: Log transformation for skewed features

pipeline = ClusterAnalysisPipeline(
    handle_missing='median',
    handle_outliers='robust',
    scaling='robust',
    log_transform_skewed=True,
    skewness_threshold=2.0
)

2. Feature Selection

• Correlation Filtering: Remove highly correlated features
• Variance Filtering: Remove low-variance features

pipeline = ClusterAnalysisPipeline(
    correlation_threshold=0.85,
    variance_threshold=0.01
)

3. Dimensionality Reduction

• PCA: Automatic component selection based on variance threshold
• t-SNE/UMAP: For 2D visualization (optional)

pipeline = ClusterAnalysisPipeline(
    pca_variance=0.9,
    pca_min_components=2
)

4. Clustering

Multiple algorithms supported:

# K-Means
pipeline = ClusterAnalysisPipeline(clustering_algorithm='kmeans', n_clusters=5)

# Gaussian Mixture Model
pipeline = ClusterAnalysisPipeline(clustering_algorithm='gmm', n_clusters=4)

# Hierarchical
pipeline = ClusterAnalysisPipeline(clustering_algorithm='hierarchical', n_clusters=3)

# DBSCAN (auto-detects clusters)
pipeline = ClusterAnalysisPipeline(clustering_algorithm='dbscan')

5. Evaluation

• Automatic optimal cluster number detection
• Multiple metrics: Silhouette, Calinski-Harabasz, Davies-Bouldin
• Elbow method support

pipeline = ClusterAnalysisPipeline(
    n_clusters=None,              # Auto-detect
    n_clusters_range=(2, 10)      # Search range
)

6. Interpretation

• Cluster profiling with feature importance
• Automatic cluster naming suggestions
• Category-based analysis

pipeline.create_profiles(category_mapping={
    'behavioral': ['sessions', 'duration', 'frequency'],
    'social': ['messages', 'friends', 'shares'],
    'engagement': ['clicks', 'likes', 'comments']
})

Visualization

If you installed viz dependencies (pip install clustertk[viz]):

from IPython.display import display

# Display multiple plots (use display() or separate cells)
display(pipeline.plot_correlation_matrix())
display(pipeline.plot_pca_variance())
display(pipeline.plot_clusters_2d(method='tsne'))
display(pipeline.plot_cluster_heatmap())
display(pipeline.plot_cluster_radar())

Jupyter usage: All plot functions return matplotlib Figure objects.

# Single plot - displays automatically
pipeline.plot_cluster_heatmap()

# Multiple plots in one cell - only last displays (standard Jupyter behavior)
# Use separate cells or display() for each:
from IPython.display import display

display(pipeline.plot_cluster_heatmap())
display(pipeline.plot_clusters_2d())
display(pipeline.plot_cluster_radar())

# Or capture for saving/manipulation
fig = pipeline.plot_cluster_heatmap()
fig.savefig('heatmap.png')

Note: When calling multiple plots in one cell, only the last one displays automatically. This is standard Python/Jupyter behavior for functions returning objects. Use display() or separate cells to show multiple plots.

Export Results

ClusterTK provides multiple ways to export your clustering results:

Export to CSV

Export cluster assignments along with original data:

# Export with original data
pipeline.export_results('results.csv', format='csv')

# Export only cluster assignments
pipeline.export_results('results.csv', format='csv', include_original=False)

The CSV will include:

• All original data columns (if include_original=True)
• cluster column with cluster assignments
• cluster_name column (if cluster naming was performed)

Export to JSON

Export comprehensive clustering metadata:

# Full export with profiles and metrics
pipeline.export_results('results.json', format='json')

# Export without profiles
pipeline.export_results('results.json', format='json', include_profiles=False)

The JSON will include:

• Cluster labels and sizes
• Cluster names (if available)
• Cluster profiles (mean feature values per cluster)
• Clustering metrics (silhouette, calinski-harabasz, etc.)
• Pipeline configuration
• Selected features list

Generate HTML Report

Create a comprehensive HTML report with visualizations:

# Full report with embedded plots
pipeline.export_report('report.html')

# Report without plots (faster, no viz dependencies needed)
pipeline.export_report('report.html', include_plots=False)

The HTML report includes:

• Clustering summary and metrics
• Cluster sizes table
• Cluster profiles heatmap table
• Embedded visualizations (if include_plots=True)
• Pipeline configuration details

Save and Load Pipeline

Save your fitted pipeline for later use:

# Save pipeline
pipeline.save_pipeline('my_pipeline.joblib')

# Load pipeline
from clustertk import ClusterAnalysisPipeline
loaded_pipeline = ClusterAnalysisPipeline.load_pipeline('my_pipeline.joblib')

# Use loaded pipeline
new_labels = loaded_pipeline.labels_
new_profiles = loaded_pipeline.cluster_profiles_

Advanced Usage

Custom Functions

You can provide custom functions for preprocessing:

def my_custom_imputer(df):
    """Custom missing value imputation logic"""
    return df.fillna(df.median())

pipeline = ClusterAnalysisPipeline(
    handle_missing=my_custom_imputer
)

Custom Clusterer

Use your own clustering implementation:

from sklearn.cluster import SpectralClustering

custom_clusterer = SpectralClustering(n_clusters=4, random_state=42)

pipeline = ClusterAnalysisPipeline(
    clustering_algorithm=custom_clusterer
)

Architecture

ClusterTK is built with a modular architecture:

clustertk/
├── preprocessing/        # Data cleaning and transformation
├── feature_selection/    # Feature filtering
├── dimensionality/       # PCA, t-SNE, UMAP
├── clustering/           # Clustering algorithms
├── evaluation/           # Metrics and optimization
├── interpretation/       # Profiling and naming
└── visualization/        # Plotting (optional)

Each module can be used independently:

from clustertk.preprocessing import MissingValueHandler
from clustertk.clustering import KMeansClustering

# Use individual components
handler = MissingValueHandler(strategy='median')
clean_data = handler.fit_transform(df)

clusterer = KMeansClustering(n_clusters=5)
labels = clusterer.fit_predict(clean_data)

Requirements

Core Dependencies

• numpy >= 1.20.0
• pandas >= 1.3.0
• scikit-learn >= 1.0.0
• scipy >= 1.7.0

Optional Dependencies

• matplotlib >= 3.4.0 (for visualization)
• seaborn >= 0.11.0 (for visualization)
• umap-learn >= 0.5.0 (for UMAP)
• hdbscan >= 0.8.0 (for HDBSCAN)

Examples

Check out the examples directory for complete notebooks:

• basic_usage.ipynb - Basic clustering workflow
• advanced_customization.ipynb - Custom preprocessing and clustering
• visualization_guide.ipynb - All visualization options
• interpretation.ipynb - Cluster profiling and interpretation

Documentation

Full documentation is available at: https://clustertk.readthedocs.io

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Citation

If you use ClusterTK in your research, please cite:

@software{clustertk,
  author = {Aleksey Veselov},
  title = {ClusterTK: A Comprehensive Toolkit for Cluster Analysis},
  year = {2024},
  url = {https://github.com/alexeiveselov92/clustertk}
}

Roadmap

• Core pipeline implementation
• Basic clustering algorithms
• Advanced clustering methods (HDBSCAN, Spectral)
• GPU support (cuML integration)
• Streaming/incremental clustering
• AutoML for hyperparameter tuning
• Web UI for interactive analysis
• Time series clustering support

Acknowledgments

ClusterTK builds upon the excellent work of:

• scikit-learn - Machine learning algorithms
• pandas - Data manipulation
• matplotlib & seaborn - Visualization

Support

• 📧 Email: [e-mail]
• 🐛 Issues: GitHub Issues
• 💬 Discussions: GitHub Discussions