ClusterExplorer

This repository contains the code for ClusterExplorer, a novel explainability tool for black-box clustering pipelines. Our approach formulates the explanation of clusters as the identification of concise conjunctions of predicates that maximize the coverage of the cluster's data points while minimizing separation from other clusters.

Explaining the results of clustering pipelines

Our approach formulates the explanation of clusters as the identification of concise conjunctions of predicates that maximize the coverage of the cluster's data points while minimizing separation from other clusters. We achieve this by reducing the problem to generalized frequent-itemsets mining (gFIM), where items correspond to explanation predicates, and itemset frequency indicates coverage. To enhance efficiency, we leverage inherent problem properties and implement attribute selection to further reduce computational costs.

Source Code

The source code is located in the cluster-explorer/src directory. This directory contains the following key components:

1. Explainer:explainer.py Generates rule-based explanations for each cluster using frequent-itemsets mining.

2. Frequent Itemset Mining:gFIM.py Contains methods for frequent itemset mining..

3. Clustering Rule Evaluation:ScoreMetrics.py AnalyzeItemsets.py Provides methods to evaluate and summarize the quality of clustering rules based on metrics such as separation error, coverage, and conciseness.

4. Binning Methods:binning_methods.py Contains methods for binning numeric attributes, including equal width, equal frequency, decision tree-based, and multiclass optimal binning techniques.

Experiment Datasets

Cluster-Explorer was evaluated using a diverse set of 98 clustering results obtained from various clustering pipelines and algorithms. The datasets used in these experiments were sourced from the UCI Machine Learning Repository and cover a wide range of data shapes and sizes.

Datasets Overview

The datasets used in the experiments include:

Clustering Pipelines

The clustering results were generated using 16 different clustering pipelines, each combining various preprocessing steps and clustering algorithms (are located in clustering_pipelines.py). The preprocessing steps included standard scaling for numeric columns, one-hot encoding for categorical data, and dimensionality reduction using PCA. The clustering algorithms used were K-Means, DBSCAN, Birch, Spectral Clustering, and Affinity Propagation.

To use this, you need to provide the datasets folder (first save the datasets in this folder) and the folder to save the pipelines results.

Running the Experiments

For running the experiments (located in cluster-explorer/experiments), you need to provide the folder of the pipelines result for BaselinesExperiment.py. The results will be saved in cluster-explorer/experiments)

Additional Experiments

This folder contains information about our attribute-selection optimization on both the explanation quality and running times. For running the experiments (located in cluster-explorer/additional_experiments), you need to provide the folder of the pipelines result for P_ValueExperiment.py. The results will be saved in cluster-explorer/additional_experiments)

Use Cases and Examples

An example of a simple use case is provided in the example_notebook.ipynb file. In this notebook we generate an explanation rules set from the 'Wine' dataset. For each cluster, the ClusterExplorer generates a set of rules that explain the common properties of the wine samples within that cluster. These rules help in understanding why certain samples are grouped together and what distinguishes one cluster from another.