CompSil

CompSil is an open-source Python package for selecting the number of clusters in unlabeled data using Composite Silhouette, an internal validation criterion that adaptively combines micro- and macro-averaged Silhouette scores across repeated subsampled clusterings.

Composite Silhouette: A Subsampling-based Aggregation Strategy

Selecting the number of clusters is a central challenge in unsupervised learning, where ground-truth labels are usually unavailable.

The standard Silhouette coefficient is one of the most widely used internal validation metrics for this task. However, its usual micro-averaged form aggregates Silhouette values over all data points, which can make the score strongly influenced by large clusters. In imbalanced datasets, this may mask poor separation or instability in smaller but meaningful groups.

A natural alternative is macro-averaging, where Silhouette values are first averaged within each cluster and then averaged across clusters. This gives every cluster equal influence, reducing the dominance of majority groups. However, macro-averaging can also overemphasize small, noisy, or under-represented clusters.

The distinction between micro- and macro-averaged Silhouette aggregation is discussed in detail in Revisiting Silhouette Aggregation by Pavlopoulos, Vardakas, and Likas. The corresponding repository is available here: https://github.com/ipavlopoulos/revisiting-silhouette-aggregation.

For users who only need direct Silhouette computation, including sample-level, micro-averaged, and macro-averaged Silhouette scores with or without approximation, see the companion Silhouette package: https://github.com/semoglou/sil_score.

These complementary failure modes create a practical dilemma:

• Micro-averaging reflects global, point-wise clustering quality but can favor majority clusters.

• Macro-averaging reflects cluster-wise balance but can overemphasize small or noisy groups.

In many applications, it is unclear in advance which view should be trusted.

CompSil addresses this issue by using the disagreement between micro- and macro-averaged Silhouette scores as a local signal for adaptive aggregation.

Composite Silhouette evaluates candidate numbers of clusters through repeated subsampled clusterings. For each candidate value of k, the method:

1. Draws multiple subsamples of the dataset.

2. Clusters each subsample.

3. Computes both micro- and macro-averaged Silhouette scores.

4. Measures their discrepancy.

5. Converts this discrepancy into a smooth convex weight.

6. Combines the two Silhouette views into a subsample-level composite score.

7. Averages the composite scores across subsamples.

For each subsample, Composite Silhouette combines the two views as:

S_mM = w * S_micro + (1 - w) * S_macro

where the weight w is determined adaptively from the normalized discrepancy between S_micro and S_macro.

This produces a single internal validation score that can be maximized over candidate values of k.

CompSil enables:

• Selection of the number of clusters without labels.

• Adaptive balancing of micro- and macro-averaged Silhouette.

• More robust cluster-count selection under size imbalance.

• Repeated subsampling for stable internal validation.

• Optional lower-confidence-bound selection using subsampling variability.

Citation

If you find this work useful, please consider citing:

Semoglou, A., Likas, A., & Pavlopoulos, J. (2027). Composite Silhouette. In Machine Learning and Knowledge Discovery in Databases. Research Track (pp. 661–678). Springer Nature Switzerland.

Presented at ECML PKDD 2026.

@InProceedings{10.1007/978-3-032-37654-1_38,
  author    = {Semoglou, Aggelos and Likas, Aristidis and Pavlopoulos, John},
  editor    = {Ceci, Michelangelo and Kocev, Dragi and Monreale, Anna and Vens, Celine and Flach, Peter and Manco, Giuseppe and Spiliopoulou, Myra and Masciari, Elio},
  title     = {Composite Silhouette},
  booktitle = {Machine Learning and Knowledge Discovery in Databases. Research Track},
  year      = {2027},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {661--678},
  isbn      = {978-3-032-37654-1},
  doi       = {10.1007/978-3-032-37654-1_38}
}

The preprint is also available on arXiv: https://arxiv.org/abs/2604.13816

Installation

Install CompSil from PyPI:

pip install compsil

Import the main class in Python as:

from compsil import CompSil

API Reference

CompSil provides a simple class-based interface for evaluating Composite Silhouette over one or more candidate numbers of clusters.

CompSil

Computes Composite Silhouette for candidate cluster counts using repeated subsampled clusterings.

CompSil(
    data,
    ground_truth=None,
    k_values=range(2, 11),
    num_samples=10,
    sample_size="auto",
    random_state=42,
    n_jobs=-1,
    eps=1e-12,
)

Inputs

• data: array-like of shape (n_samples, n_features)
  Input data matrix.

• ground_truth: int or None, default None
  Optional reference number of clusters.
  Used only for visualization.

• k_values: iterable of int or int, default range(2, 11)
  Candidate number or candidate numbers of clusters to evaluate.

• num_samples: int, default 10
  Number of subsamples used for each candidate value of k.

• sample_size: int, float, None, or "auto", default "auto"
  Subsample size used in each repeated clustering.

  • If int, it is interpreted as the absolute subsample size.
  • If float in (0, 1], it is interpreted as a fraction of the dataset size.
  • If None or "auto", the subsample size is selected automatically from the dataset size and the largest candidate value of k.

• random_state: int, default 42
  Base random seed used for reproducible subsampling and clustering.

• n_jobs: int, default -1
  Number of parallel jobs used during evaluation.

• eps: float, default 1e-12
  Numerical stability constant used when normalizing micro–macro discrepancies.

evaluate

Evaluates Composite Silhouette over all candidate values of k.

model.evaluate()

After calling evaluate, the results are stored in:

model.results_df

The results table contains:

• k: candidate number of clusters.
• avg S_micro: average micro-averaged Silhouette across subsamples.
• avg S_macro: average macro-averaged Silhouette across subsamples.
• w_micro: average adaptive weight assigned to the micro view.
• S_mM: Composite Silhouette score.
• std S_mM: standard deviation of subsample-level composite scores.
• se S_mM: standard error of the Composite Silhouette estimate.
• LCB S_mM: lower-confidence-bound score, computed as S_mM - se S_mM.
• B_eff: number of valid subsampling trials.
• sample_size: resolved subsample size.
• sample_fraction: resolved subsample fraction.

get_optimal_k

Returns the selected number of clusters.

model.get_optimal_k(use_lcb=False)

Inputs

• use_lcb: bool, default False
  If False, selects the k that maximizes S_mM.
  If True, selects the k that maximizes LCB S_mM.

Returns

• optimal_k: int
  Selected number of clusters.

get_results_dataframe

Returns the results as a pandas DataFrame indexed by k.

results = model.get_results_dataframe()

Returns

• results: pandas DataFrame
  Table containing the Composite Silhouette results for all candidate values of k.

plot_results

Plots the Composite Silhouette curve together with the subsample-averaged micro- and macro-averaged Silhouette curves.

model.plot_results()

If ground_truth was provided, it is shown as a vertical reference line.

Quick Start

This example creates a simple synthetic dataset with five Gaussian clusters, evaluates candidate values of k, and selects the number of clusters using Composite Silhouette.

from sklearn.datasets import make_blobs
from sklearn.preprocessing import StandardScaler
from compsil import CompSil

# Create a simple synthetic dataset
X, y = make_blobs(
    n_samples=1000,
    centers=5,
    n_features=10,
    cluster_std=1.5,
    random_state=42,
)

# Standardize the data
X = StandardScaler().fit_transform(X)

# Initialize Composite Silhouette
model = CompSil(
    data=X,
    ground_truth=5,
    k_values=range(2, 11),
    num_samples=10,
    sample_size="auto",
    random_state=0,
    n_jobs=-1,
)

# Evaluate all candidate k values
model.evaluate()

# Select the number of clusters
best_k = model.get_optimal_k()

print("Selected k:", best_k)

# Inspect the full results table
results = model.get_results_dataframe()
print(results)

# Plot the Composite Silhouette curve
model.plot_results()

The S_mM column in the results table contains the Composite Silhouette score for each candidate number of clusters. The selected number of clusters is the value of k that maximizes S_mM.

CompSil can also be used to evaluate a single candidate number of clusters. In this case, pass an integer to k_values.

# Evaluate a single candidate k
model = CompSil(
    data=X,
    k_values=5
)

model.evaluate()

# Composite Silhouette score for k=5
print("Composite Silhouette score:", model.score_)

# Full results table
results = model.get_results_dataframe()
print(results)

When a single value of k is evaluated, model.score_ stores the corresponding Composite Silhouette score.

Acknowledgments

This work was supported by Archimedes Research Unit, Athena Research Center.

Contact

For questions or feedback about Composite Silhouette, feel free to contact me at a.semoglou@outlook.com.

License

This project is licensed under the MIT License.

Links

• Published paper: SPRINGER ECML PKDD
• DOI: 10.1007/978-3-032-37654-1_38
• Preprint: arXiv:2604.13816
• Package: PyPI
• Presentation: ECML PKDD 2026