dimkit

A unified Python library for dimensionality reduction with a consistent, sklearn-compatible API.

dimkit wraps 10+ reduction methods behind a single interface so you can swap algorithms without rewriting code, compare them side-by-side, and get practical guidance on when to use each one.

---

Features

• Unified API — every reducer implements fit, transform, fit_transform, get_params, set_params, summary
• 10 reduction methods — PCA, TruncatedSVD, FactorAnalysis, FastICA, t-SNE, UMAP*, Isomap, LLE, MDS, SpectralEmbedding, Autoencoder*
• Honest about limitations — methods like t-SNE that don't support transform() raise TransformNotSupportedError with clear guidance
• Compare reducers — compare_reducers() runs multiple methods and collects runtime, trustworthiness, EVR, and reconstruction error
• Recommender — recommend_reducer() gives rule-based advice for your use case
• Visualization utilities — scatter plots, scree plots, cumulative variance, side-by-side comparisons
• Evaluation metrics — trustworthiness, continuity, pairwise distance correlation, reconstruction MSE
• Lightweight core — UMAP and the autoencoder are optional extras

* Requires optional dependencies.

---

Installation

# Core (PCA, SVD, FA, ICA, t-SNE, Isomap, LLE, MDS, SpectralEmbedding)
pip install dimkit

# With UMAP support
pip install dimkit[umap]

# With Plotly (interactive plots)
pip install dimkit[plot]

# With PyTorch autoencoder
pip install dimkit[deep]

# Everything
pip install dimkit[all]

# Development
pip install dimkit[dev]

Install from source (editable)

git clone https://github.com/example/dimkit.git
cd dimkit
pip install -e ".[dev]"

---

Quick Start

from sklearn.preprocessing import StandardScaler
from dimkit import PCAReducer
from dimkit.datasets import load_iris

X, y, _ = load_iris()
X_scaled = StandardScaler().fit_transform(X)

reducer = PCAReducer(n_components=2)
X_2d = reducer.fit_transform(X_scaled)

print(reducer.explained_variance_ratio_)
# [0.729 0.228]

---

Supported Methods

Class	Method	Family	Deterministic	Supports transform()
PCAReducer	PCA	Linear	Yes	Yes
TruncatedSVDReducer	TruncatedSVD / LSA	Linear	Yes	Yes
FactorAnalysisReducer	Factor Analysis	Linear	Yes	Yes
FastICAReducer	FastICA	Linear	No	Yes
TSNEReducer	t-SNE	Manifold	No	No
UMAPReducer	UMAP	Manifold	No	Yes
IsomapReducer	Isomap	Manifold	Yes	Yes
LLEReducer	LLE	Manifold	Yes	Yes
MDSReducer	MDS	Manifold	No	No
SpectralReducer	SpectralEmbedding	Manifold	No	No
AutoencoderReducer	Autoencoder	Deep	No	Yes

---

Comparing Multiple Methods

from dimkit import PCAReducer, TSNEReducer, IsomapReducer
from dimkit.compare import compare_reducers
from sklearn.preprocessing import StandardScaler
from dimkit.datasets import load_digits

X, y, _ = load_digits(n_class=5)
X_scaled = StandardScaler().fit_transform(X)

report = compare_reducers(
    X_scaled,
    reducers=[
        PCAReducer(n_components=2),
        TSNEReducer(n_components=2, perplexity=30, random_state=42),
        IsomapReducer(n_components=2, n_neighbors=10),
    ],
    compute_trustworthiness=True,
)

print(report.summary())
print(f"Fastest:              {report.fastest().reducer_name}")
print(f"Best trustworthiness: {report.best_trustworthiness().reducer_name}")

---

Visualizing Embeddings

from dimkit.visualization import plot_embedding, plot_scree, plot_comparison

# Single embedding
ax = plot_embedding(X_2d, labels=y, title="PCA — Digits")

# Scree plot
ax = plot_scree(reducer)

# Side-by-side comparison
fig = plot_comparison(report.results, labels=y)

---

Getting a Recommendation

from dimkit.compare import recommend_reducer

rec = recommend_reducer(
    goal="visualization",       # or "preprocessing", "clustering", "manifold"
    data_shape=(5000, 100),
    sparse=False,
    interpretability_needed=False,
)
print(rec)
# Recommendation: UMAP
# Alternatives:   t-SNE, PCA
# Reasoning: UMAP produces visually strong 2D/3D embeddings...

---

Handling Unsupported Operations

dimkit raises clear, informative errors rather than silently failing:

from dimkit import TSNEReducer
from dimkit.exceptions import TransformNotSupportedError

reducer = TSNEReducer(n_components=2, random_state=42)
reducer.fit_transform(X_scaled)

try:
    reducer.transform(X_scaled[:10])  # t-SNE does not support this
except TransformNotSupportedError as e:
    print(e)
    # 't-SNE' does not support transform() on new data. Use fit_transform()
    # on the full dataset instead. For out-of-sample projection, use UMAPReducer.

---

Preprocessing Helpers

from dimkit.utils.preprocessing import (
    standardize_features,
    scale_features,
    check_for_missing,
    impute_missing,
)

# Check for NaN before processing
info = check_for_missing(X)
if info["has_missing"]:
    X = impute_missing(X, strategy="mean")

# Scale (returns array + fitted scaler)
X_scaled, scaler = scale_features(X, method="standard")

---

Evaluation Metrics

from dimkit.metrics import (
    trustworthiness,
    continuity,
    reconstruction_mse,
    pairwise_distance_correlation,
)

trust = trustworthiness(X_original, X_embedded, n_neighbors=5)
cont  = continuity(X_original, X_embedded, n_neighbors=5)
corr  = pairwise_distance_correlation(X_original, X_embedded)
mse   = reconstruction_mse(X_original, X_reconstructed)

---

Running Tests

cd dimkit
pip install -e ".[dev]"
pytest

---

Limitations and Cautions

t-SNE

• Cluster sizes and inter-cluster distances in t-SNE plots are not interpretable
• Results vary significantly between runs — always set random_state
• Perplexity strongly affects visual output; try several values (5, 30, 50)
• Not suitable for large datasets (>50k samples)

UMAP

• Global distances are more meaningful than t-SNE but still approximate
• n_neighbors controls local/global balance — tune it
• Always set random_state

General

• Dimensionality reduction for visualization does not guarantee useful features for modeling
• Preprocessing matters: scale your data before applying most reducers
• Nonlinear embeddings can distort both local and global relationships
• Visual cluster separation does not imply that clusters are separable in feature space

---

Project Structure

dimkit/
  pyproject.toml
  src/
    dimkit/
      base.py           # Abstract BaseReducer
      exceptions.py     # Custom exceptions
      reducers/         # One file per method
      compare/          # compare_reducers, recommend_reducer
      metrics/          # trustworthiness, reconstruction MSE, runtime
      visualization/    # scatter, scree, comparison plots
      datasets/         # sample datasets (Iris, Digits, blobs, Swiss roll)
      utils/            # validation, preprocessing helpers
  tests/                # pytest suite
  examples/             # runnable example scripts
  docs/                 # method_guide.md, examples.md

---

Future Enhancements

• Kernel PCA (KernelPCAReducer)
• Non-negative Matrix Factorisation (NMF)
• PHATE (Potential of Heat-diffusion for Affinity-based Trajectory Embedding)
• Interactive Plotly-based comparison dashboard
• Variational Autoencoder (VAE) reducer
• Pipeline integration with sklearn Pipeline
• Hyperparameter tuning helpers (grid search over perplexity / n_neighbors)
• CSV/DataFrame loading convenience wrapper

---

License

MIT — see LICENSE.