whc 1.0.0

Wormhole Hyperconnections — Drop-in replacement for residual connections

Wormhole Hyperconnections (WHC)

A drop-in replacement for the standard residual connection x = x + layer(x).

WHC replaces the single residual path with n parallel lanes and learnable mixing matrices, delivering:

• 6–15% faster training than standard residual
• Lower final loss on benchmark tasks
• Proven stability — spectral norm ≤ 1
• 82 parameters for the mixing matrix (vs mHC's 32,768)
• 2× faster than mHC in wall-clock time

---

Quick Start

from whc import WormholeHyperconnection

whc = WormholeHyperconnection(
    dim=512,
    expansion_rate=4,
    num_layers=12,
)

x = whc.expand(x)                      # (B, T, dim) -> (B, T, n, dim)
for i, layer in enumerate(transformer_blocks):
    x = whc(x, layer, layer_idx=i)      # replaces `x = x + layer(x)`
x = whc.reduce(x)                      # (B, T, n, dim) -> (B, T, dim)

---

Installation

From PyPI (Recommended)

pip install whc

From Source

git clone https://github.com/fardinsabid/wHC.git
cd wHC
pip install -e .

Requirements

• Python >= 3.8
• PyTorch >= 2.0.0

---

Key Features

Feature	Description
Drop-in replacement	Replace x = x + layer(x) with x = whc(x, layer=layer)
n parallel lanes	Instead of 1 fixed path
Learnable mixing	Network learns how lanes interact
Proven stability	Spectral norm ≤ 1 guarantees no explosion
No iterations	Closed-form kernel, no Sinkhorn-Knopp
Minimal overhead	82 parameters for the mixing matrix
GPU ready	Runs on CUDA, CPU, MPS

---

Benchmarks

Signal Gain vs Depth

Method	Mean Final Gain	Max Final Gain
Unconstrained HC	9.54×10⁵	4.77×10⁶
mHC (DeepSeek)	0.42	0.68
WHC	0.48	0.63

WHC and mHC stay bounded; unconstrained HC explodes.

Training Loss

Method	Final Loss	Time (s)
Standard Residual	0.3765	0.91
mHC-static	0.6966	11.01
WHC	0.3590	5.54

WHC achieves lower loss and is 2× faster than mHC.

Parameter Count

Component	Parameters
WormholeKernel (n=8)	82
mHC H_res generator (dim=512, n=8)	32,768

WHC uses ~400× fewer parameters for the mixing matrix.

---

Usage

1. Basic Usage (Manual Integration)

from whc import WormholeHyperconnection

# Create WHC wrapper
whc = WormholeHyperconnection(dim=512, expansion_rate=4, num_layers=12)

# Forward pass
x = whc.expand(x)
for i, layer in enumerate(transformer_blocks):
    x = whc(x, layer, layer_idx=i)
x = whc.reduce(x)

2. Single Layer

whc = WormholeHyperconnection(dim=512, expansion_rate=4, num_layers=1)

x = whc.expand(x)
x = whc(x, layer, layer_idx=0)
x = whc.reduce(x)

3. With Pre-computed Layer Output

layer_out = layer(x)
x = whc(x, layer_out=layer_out)

4. Custom Configuration

whc = WormholeHyperconnection(
    dim=768,
    expansion_rate=8,           # Number of parallel lanes
    num_layers=24,              # Number of layers in your stack
    manifold_dim=3,             # Dimension of the wormhole manifold
    shared_kernel=False,        # Share kernel across layers?
)

---

Examples

Example	Description	Run
simple_mlp.py	MLP with WHC replacing residuals	python examples/simple_mlp.py
transformer_whc.py	Transformer with WHC replacing residuals	python examples/transformer_whc.py
resnet_whc.py	ResNet with WHC replacing residuals	python examples/resnet_whc.py

---

Tests

pytest tests/ -v

Test File	What It Tests
test_kernel.py	Kernel stability, spectral norm ≤ 1
test_whc.py	Shape, forward pass, parameters
test_gradients.py	Gradient flow, no NaNs, no explosion

All 21 tests passed ✅

---

Research Paper

The full mathematical derivation, stability proof, and experimental results are in:

📄 papers/whc.pdf

Key contributions:

• Spectral normalization (closed-form, no iteration)
• Wormhole kernel (physics-inspired, interpretable)
• 82 parameters for H_res (vs mHC's 32,768)
• 2× faster than mHC
• Better loss than standard residual

---

Project Structure

whc/
├── whc.py                          # Core implementation
├── README.md                       # This file
├── LICENSE                         # MIT License
├── setup.py                        # Package installer
├── pyproject.toml                  # Build config
├── requirements.txt                # Dependencies
├── examples/
│   ├── simple_mlp.py               # MLP with WHC
│   ├── transformer_whc.py          # Transformer with WHC
│   └── resnet_whc.py               # ResNet with WHC
├── papers/
│   └── whc.pdf                     # Research paper
├── assets/
│   ├── whc_stability.png           # Stability comparison
│   └── whc_training.png            # Training loss curves
└── tests/
    ├── test_kernel.py              # Kernel unit tests
    ├── test_whc.py                 # WHC unit tests
    └── test_gradients.py           # Gradient stability tests

---

Citation

If you use WHC in your research, please cite:

@misc{sabid2026whc,
  author = {Fardin Sabid},
  title = {Wormhole Hyperconnections: A Physics-Inspired Framework for Stable Deep Residual Learning},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/fardinsabid/wHC}}
}

---

License

MIT License — see LICENSE for details.

---

Author

Fardin Sabid

• GitHub: @fardinsabid
• Research: Deep Learning Optimization, Physics-Inspired Architectures

---

Acknowledgments

WHC builds on the foundation of:

• He et al. (2016) — Residual connections
• Zhu et al. (2025) — Hyper-Connections (HC)
• Xie et al. (2025/2026) — Manifold-Constrained Hyper-Connections (mHC)
• Kipf & Welling (2017) — Graph Convolutional Networks

---

Star Us

If you find WHC useful, please ⭐ star the repository!

---

The standard residual was 2016. WHC is 2026.