kan-mlp-hybrid 0.1.1

Hybrid Kolmogorov-Arnold Multi-Layer Perceptrons for PyTorch

🧠 KAN-MLP Hybrid Architectures

Hybrid Kolmogorov-Arnold Multi-Layer Perceptrons for PyTorch

📖 Overview

kan-mlp-hybrid is a PyTorch library that bridges the gap between the compositional efficiency of Multi-Layer Perceptrons (MLPs) and the univariate approximation power of Kolmogorov-Arnold Networks (KANs).

While standard MLPs use fixed activation functions (like ReLU or SiLU) and learnable linear weights, and pure KANs use learnable spline-based activations on edges with no linear weights, Hybrid KAN-MLPs combine both. This allows the network to learn arbitrary univariate activation functions per neuron while maintaining the efficient high-dimensional feature routing of dense linear layers.

Key Features

• 🚀 Drop-in PyTorch Modules: Easily replace nn.Linear or nn.Sequential with our hybrid layers.
• 📈 Superior Parameter Efficiency: Decouples the spline grid size from the quadratic width term, avoiding the $O(N^2LG)$ parameter explosion of pure KANs.
• 🧠 Multiple Architectures: Includes LA-MLP (Learnable Activation), PPH-Net (Parallel Pathway), and MoE-KAN-MLP (Mixture-of-Experts).
• 📊 Benchmarked: Proven to outperform standard MLPs on mixed-characteristic functions, tabular data, and time-series forecasting.

---

🛠️ Installation

You can install the package directly from PyPI:

pip install kan-mlp-hybrid

Or install from source for the latest updates:

git clone https://github.com/useCallback/kan-mlp-hybrid.git
cd kan-mlp-hybrid
pip install -e .

---

🚀 Quick Start

Using the Learnable Activation MLP (LA-MLP) is as simple as using a standard PyTorch MLP.

import torch
from kan_mlp_hybrid import LAMLP

# Initialize an LA-MLP model
# in_features: 5, hidden layers: [64, 64], out_features: 1
model = LAMLP(
    in_features=5,
    hidden_features=[64, 64],
    out_features=1,
    num_layers=3,
    grid_size=5,       # Number of B-spline grid intervals
    spline_order=3     # Order of the piecewise polynomial
)

# Forward pass
x = torch.randn(32, 5)
y = model(x)

print(y.shape) # Output: torch.Size([32, 1])

Available Architectures

1. LAMLP: Replaces fixed activation functions in standard MLPs with learnable, spline-based univariate functions. Best overall performance and efficiency.
2. PPHNet: Processes inputs through parallel KAN and MLP pathways, fusing them via a learned gating mechanism.
3. MoEKANMLP: A sparsely gated architecture containing a heterogeneous pool of KAN and MLP experts.
4. PureKAN: A baseline implementation of a pure Kolmogorov-Arnold Network.

---

📓 Interactive Colab Notebooks

We provide ready-to-run Google Colab notebooks demonstrating how to use kan-mlp-hybrid across various domains:

• Tabular Data (Classification & Regression): Breast Cancer & California Housing datasets using LAMLP and PPHNet.
• Computer Vision: MNIST digit classification using MoEKANMLP.
• Natural Language Processing: AG News text classification using TF-IDF and PureKAN.

---

🔬 Scientific Background

This library is based on the theoretical synthesis of the Universal Approximation Theorem (underpinning MLPs) and the Kolmogorov-Arnold representation theorem (underpinning KANs).

By utilizing the LALinear layer, the linear transformations $W$ efficiently handle high-dimensional basis projection, while the learnable activations $\phi$ achieve exponential convergence on the projected 1D manifolds.

For a deep dive into the mathematical formulations, parameter complexity analysis, and architectural brainstorming that led to this package, please refer to the RESEARCH_NOTES.md and PAPER_DRAFT.md files in the GitHub repository.

---

👨‍💻 Author

Khalfoun Mohamed El Mehdi

• GitHub: @useCallback
• PyPI: FastFourierTransform

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.