laminarnet 0.1.0

A novel neural architecture merging SSM, RNN, and Hierarchical processing.

LaminarNet: Structured Orthogonal State Space Sequence Model

A Next-Generation Neural Architecture for Long-Context Sequence Modeling

---

🌊 Overview

LaminarNet is a novel deep learning architecture designed to overcome the limitations of traditional Transformers in handling long sequences. By fusing principles from State Space Models (SSMs), Recurrent Neural Networks (RNNs), and Hierarchical Processing, LaminarNet achieves $O(N)$ inference complexity while maintaining the parallel training benefits of Transformers.

At its core, LaminarNet introduces three groundbreaking mechanisms:

1. Geometric Drift Fields (GDF): Parallelizable rotation-based state evolution that replaces standard RNN loops with prefix sums.
2. Cross-Stratum Routing (CSR): A bidirectional, multi-resolution information exchange system that allows different layers of abstraction to communicate effectively.
3. Phase Mesh Encoding (PME): A stabilized coupled oscillator system for position encoding, offering superior extrapolation capabilities compared to Rotary Embeddings (RoPE).

🚀 Key Features

• Linear Complexity: Scales linearly with sequence length during inference, making it ideal for long-context applications.
• Hierarchical Structure: Processes information at multiple resolutions (strata) simultaneously, capturing both local details and global context.
• Parallel Training: Utilizes prefix scan algorithms to enable efficient parallel training on GPUs.
• Stabilized Dynamics: Incorporates gated retention and phase clipping to ensure stable training dynamics even at depth.

📦 Installation

Install LaminarNet directly from PyPI:

pip install laminarnet

🛠️ Usage

Basic Example

import torch
from laminarnet import LaminarNet, LaminarNetConfig

# Initialize configuration
config = LaminarNetConfig(
    vocab_size=32000,
    d_model=256,
    n_layers=6,
    seq_len=2048,
    n_strata=2
)

# Create model
model = LaminarNet(config)

# Forward pass
inputs = torch.randint(0, 32000, (1, 128))  # (Batch, SeqLen)
logits = model(inputs)

print(f"Output shape: {logits.shape}")
# Output: torch.Size([1, 128, 32000])

Advanced Configuration

You can customize the depth, width, and hierarchical structure of the network:

config = LaminarNetConfig(
    d_model=512,
    n_heads=8,
    n_layers=12,
    d_ff=2048,
    n_strata=3,            # 3 levels of hierarchy
    strata_ratios=(1, 4, 16), # Compression ratios for each stratum
    n_oscillators=32,      # Number of phase oscillators
    dropout=0.1
)

🔬 Architecture Deep Dive

1. Geometric Drift Fields (GDF)

Unlike traditional RNNs that rely on matrix multiplications for state updates, GDF employs a rotation-based mechanism in a high-dimensional vector space. By using parallel prefix sums (cumulative rotations), GDF achieves the sequential modeling power of an RNN with the parallel efficiency of a Transformer.

# Conceptual GDF Update
theta = tanh(Project(x))
state = state * rotation(theta) + input

2. Cross-Stratum Routing (CSR)

CSR enables information to flow between "fast" strata (processing high-frequency details) and "slow" strata (processing global context). This mimics the biological plausibility of neural oscillation coupling in the brain.

3. Phase Mesh Encoding (PME)

Replacing static positional embeddings, PME uses a system of coupled oscillators to encode position as a dynamic state. This allows the model to generalize to sequence lengths far beyond those seen during training.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Developed with ❤️ by Unan