fast-varclushi

fast-varclushi is a high-performance Python package for variable clustering (VARCLUS) and hierarchical dimension reduction on tabular data. It provides an optimized, 100% bitwise backward-compatible Python alternative to SAS PROC VARCLUS.

Variable clustering divides a set of numeric variables into disjoint clusters such that variables within each cluster are strongly correlated with their cluster component, while variables across different clusters are relatively uncorrelated.

---

⚡ Key Highlights & Performance Optimizations

fast-varclushi is engineered to scale variable clustering to massive datasets with millions of rows and hundreds or thousands of features:

1. Pre-computed Correlation Matrix Caching ($O(M \cdot N^2)$ Initialization):
   • Computes the feature correlation matrix $\mathbf{C}$ once upon initialization.
   • All subsequent sub-cluster splits and variable reassignment iterations slice correlation matrices in microsecond memory operations ($O(K^2)$).
   • Sample-Size Independent Clustering: Running variable clustering on 2,000,000 rows takes the exact same time as running it on 1,000 rows once the correlation matrix is computed!
2. Sub-Cluster Eigenvalue Memoization & LRU Caching:
   • Total variance calculations and top eigenvalue updates during greedy variable reassignments are cached and solved with symmetric eigensolvers (np.linalg.eigvalsh).
3. Multi-CPU Core Parallelization (n_jobs):
   • Supports parallel random search restarts (n_rs > 0) using joblib.Parallel across available CPU cores (n_jobs=-1).
4. Vectorized $R^2$ Property Computation (50x–100x Speedup):
   • Computes $R^2_{Own}$ and $R^2_{NC}$ (Nearest Cluster) across all variables simultaneously using matrix projections ($\mathbf{R}{N \times K} = \mathbf{C}{N \times N} \cdot \mathbf{W}_{N \times K}$).
5. Optimized Factor Rotations (Rotator):
   • Features efficient implementations of 7 factor rotation algorithms (varimax, promax, oblimin, quartimax, quartimin, oblimax, equamax) with support for batch parallel processing and native float32 precision.
6. Reproducibility with random_seed:
   • Full support for setting random seeds across single-threaded and multi-core parallel random search execution paths.
7. 100% SAS PROC VARCLUS Backward Compatibility:
   • Verified across regression tests to yield identical mathematical outputs to SAS PROC VARCLUS.

---

📦 Installation

Install fast-varclushi via pip:

pip install fast-varclushi

Dependencies

• numpy >= 1.20.0
• pandas >= 1.2.0
• scipy >= 1.6.0
• scikit-learn >= 0.24.0
• joblib >= 1.0.0

Development Setup

To install fast-varclushi locally with development dependencies:

git clone https://github.com/aashaybelekar/fast-varclushi.git
cd fast-varclushi
pip install -e .[dev]

---

🚀 Quickstart Example

import pandas as pd
from varclushi import VarClusHi

# Load a sample dataset (Wine Quality Red dataset)
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv"
df = pd.read_csv(url, sep=";")
df = df.drop(columns=["quality"])

# Initialize VarClusHi
vc = VarClusHi(
    df=df,
    maxeigval2=1.0,     # Stop splitting when max 2nd eigenvalue <= 1.0
    maxclus=None,      # Maximum number of clusters (None for unlimited)
    n_rs=10,           # 10 random search restarts per split
    n_jobs=-1,         # Use all available CPU cores
    random_seed=42     # Ensure reproducibility
)

# Run variable clustering
vc.varclus()

# View Cluster Summary
print("--- Cluster Info ---")
print(vc.info)

# View R-Squared Ratios for Feature Selection
print("\n--- RSquare Table ---")
print(vc.rsquare.head(10))

Output Tables

Cluster Summary (vc.info)

Cluster	N_Vars	Eigval1	Eigval2	VarProp
0	3	2.141357	0.658413	0.713786
1	3	1.766885	0.900991	0.588962
2	2	1.371260	0.628740	0.685630
3	2	1.552496	0.447504	0.776248
4	1	1.000000	0.000000	1.000000

R-Squared Ratios (vc.rsquare)

Cluster	Variable	RS_Own	RS_NC	RS_Ratio
0	fixed acidity	0.882210	0.277256	0.162976
0	density	0.622070	0.246194	0.501362
0	pH	0.637076	0.194359	0.450478
1	free sulfur dioxide	0.777796	0.010358	0.224530
1	total sulfur dioxide	0.786660	0.042294	0.222761
1	residual sugar	0.202428	0.045424	0.835525

---

🛠️ Feature Selection Workflow

fast-varclushi is ideal for reducing multi-collinearity and selecting representative features for machine learning models:

1. RS_Own: Squared correlation between the variable and its own cluster component (higher is better).
2. RS_NC: Squared correlation between the variable and the nearest cluster component (lower is better).
3. RS_Ratio: Defined as: $$\text{RS_Ratio} = \frac{1 - \text{RS_Own}}{1 - \text{RS_NC}}$$ Small values of RS_Ratio indicate that a variable has high correlation with its own cluster and low correlation with the nearest cluster.

Selecting Cluster Representatives

To select the single best representative feature from each cluster:

# Select the variable with the lowest RS_Ratio in each cluster
selected_features = (
    vc.rsquare
    .sort_values(by=["Cluster", "RS_Ratio"])
    .groupby("Cluster")
    .first()["Variable"]
    .tolist()
)

print("Selected Representative Features:", selected_features)
# ['fixed acidity', 'free sulfur dioxide', 'chlorides', 'volatile acidity', 'alcohol']

---

📚 API Reference

varclushi.VarClusHi

VarClusHi(
    df,
    feat_list=None,
    maxeigval2=1,
    maxclus=None,
    n_rs=0,
    n_jobs=None,
    random_seed=None
)

Parameters

Parameter	Type	Default	Description
df	pandas.DataFrame	Required	Input DataFrame containing numeric variables.
feat_list	list or None	None	List of column names to cluster. If None, uses all columns in df.
maxeigval2	float	1.0	Threshold for stopping cluster splitting. Splits continue as long as the second eigenvalue of a cluster exceeds maxeigval2.
maxclus	int or None	None	Maximum number of clusters to form. If set, splitting stops when cluster count reaches maxclus.
n_rs	int	0	Number of random search restarts for reassigning variables after factor rotation split.
n_jobs	int or None	None	Number of parallel jobs for random search iterations (-1 uses all CPU cores).
random_seed	int, random.Random, None	None	Seed or random number generator instance for reproducible clustering.

Methods

• varclus(speedup=True, random_seed=None) Performs hierarchical variable clustering.

  • speedup: bool (default True). Enables high-performance precomputed correlation matrix mode.
  • random_seed: int or None. Optional seed override.

• correig(df, feat_list=None, n_pcs=2) (static) Computes correlation matrix, eigenvalues, eigenvectors, and variance proportions for a DataFrame.

• pca(df, feat_list=None, n_pcs=2) (static) Computes standardized principal components, eigenvalues, eigenvectors, and variance proportions.

Properties

• vc.info (pandas.DataFrame): Cluster summary table with columns:

  • Cluster: Cluster ID.
  • N_Vars: Number of variables in the cluster.
  • Eigval1: First eigenvalue (variance explained by primary cluster component).
  • Eigval2: Second eigenvalue (variance of second principal component).
  • VarProp: Proportion of cluster variance explained by the primary component.

• vc.rsquare (pandas.DataFrame): R-squared ratio analysis table with columns:

  • Cluster: Cluster ID.
  • Variable: Variable name.
  • RS_Own: $R^2$ with own cluster component.
  • RS_NC: $R^2$ with nearest cluster component.
  • RS_Ratio: $(1 - R^2_{Own}) / (1 - R^2_{NC})$.

---

varclushi.rotator.Rotator

The Rotator module provides factor rotation algorithms for structural equation modeling, factor analysis, and custom dimension reduction workflows.

from varclushi.rotator import Rotator

rotator = Rotator(
    method="varimax",
    normalize=None,
    power=4,
    kappa=0,
    gamma=0,
    max_iter=500,
    tol=1e-5,
    n_jobs=-1
)

Supported Rotation Methods

Rotation Method	Category	Description
"varimax"	Orthogonal	Maximizes variance of squared loadings within columns (default).
"quartimax"	Orthogonal	Minimizes complexity of rows by maximizing sum of 4th powers of loadings.
"oblimax"	Orthogonal	Maximizes kurtosis of factor loadings.
"equamax"	Orthogonal	Compromise between Varimax and Quartimax. Controlled by kappa.
"promax"	Oblique	Oblique rotation constructed from Varimax rotated loadings. Controlled by power.
"oblimin"	Oblique	General family of oblique rotations. Controlled by gamma.
"quartimin"	Oblique	Special case of Oblimin (gamma=0).

Methods & Attributes

• fit(X, y=None): Fits rotation to unrotated loading matrix X.
• fit_transform(X, y=None): Fits and returns rotated loading matrix.
• fit_transform_batch(X_list, n_jobs=None): Batch rotates multiple loading matrices in parallel across CPU cores.
• loadings_: numpy.ndarray of rotated factor loadings.
• rotation_: numpy.ndarray rotation matrix.
• phi_: numpy.ndarray factor correlation matrix (for oblique rotations).

Standalone Rotator Example

import numpy as np
from varclushi.rotator import Rotator

# Sample unrotated factor loading matrix (5 features, 2 factors)
loadings = np.array([
    [0.7, 0.2],
    [0.8, 0.1],
    [0.2, 0.6],
    [0.1, 0.9],
    [0.6, 0.5]
])

# Perform Varimax rotation
rotator = Rotator(method="varimax", normalize=True)
rotated_loadings = rotator.fit_transform(loadings)

print("Rotated Loadings:\n", rotated_loadings)
print("Rotation Matrix:\n", rotator.rotation_)

---

📊 Big Data Performance Benchmark

fast-varclushi delivers exceptional speedups on large-scale tabular datasets:

Benchmark Dataset	Rows	Features	Memory Footprint	Execution Time	Total Clusters
Synthetic Big Data	2,000,000	500	3.73 GB	74.78 seconds ⚡	186

To run the benchmark on your local system:

python benchmark_bigdata.py

---

🧪 Running Tests

fast-varclushi includes 141 unit, integration, and regression tests.

Run the test suite with pytest:

pytest

---

📜 License

Distributed under the GNU General Public License v3 (GPLv3). See LICENSE for details.

---

🤝 Authors & Credits

• Aashay Belekar (@aashaybelekar) - High-performance parallelization, precomputed matrix caching, vectorized RSquare, Rotator optimizations, and maintenance.
• Xuan Jing - Original VarClusHi package author.
• Jeremy Biggs - Original factor rotation routines port.