Ultra-Fused Transformer v6.1 — SDLA with DeepSeek-MLA Compression

A high-performance transformer library optimized for hardware constraints, featuring Selective Differential Linear Attention (SDLA) integrated with DeepSeek-MLA style low-rank compression and YaRN context window extension.

This project targets the key memory bottleneck of KV-Cache in long-context models while retaining precision and noise-filtering capabilities via differential mechanics.

Core Innovations in v6.1

1. DeepSeek-MLA Style Low-Rank Compression

Rather than caching raw high-dimensional KV vectors, v6.1 compresses the entire attention space down to a shared latent bottleneck:

• Latent Projection: Compresses activation vectors from $d_{model} \rightarrow d_{model} / \text{compression_ratio}$ prior to attention computation.
• Decoupled RoPE: Separates content representations from positional embeddings (akin to DeepSeek-V3), ensuring position-dependent keys don't break the low-rank structure.
• Efficiency: Delivers a 4x to 16x reduction in memory footprint compared to standard Multi-Head Attention (MHA).

2. YaRN / SuFT Long-Context Extension

Enables 4x to 8x context length extension out-of-the-box without requiring resource-intensive retraining:

• Implements NTK-by-parts interpolation paired with attention temperature scaling.
• Preserves perplexity when scaling up sequence lengths during inference.
• Mathematical reference based on: YaRN: Efficient Context Window Extension.

3. Learnable Lambda + RMSNorm Stabilization

Deep differential networks often suffer from gradient explosion or vanishing signals. To stabilize deep layers, we introduce:

• Per-Head Learnable $\lambda$: Every individual attention head learns its own optimized denoising strength.
• Layer-Scale $\lambda$: A global learnable multiplier across stacked layers.
• Post-Differential RMSNorm: Re-normalizes variance back to $\approx 1.0$ immediately following the differential operation: $$\text{Output} = Q_1K_1^T - \lambda \cdot (Q_2K_2^T)$$

4. 3-Level Entropy-Based Dynamic Router

Optimizes Feed-Forward Network (FFN) compute paths per-token based on attention entropy proxies:

• Level 1 (Early Exit): Routes through a lightweight FFN branch only, saving up to 80% of standard compute.
• Level 2 (Alpha Blend): Executes a weighted linear combination of the lightweight and full FFN blocks.
• Level 3 (Full Compute): Engages both branches at maximum capacity for structurally complex tokens.

5. Triton Hardware Acceleration & Quantization

• Fused Triton Kernel: Collapses RMSNorm, QKV Projection, and Differential preparation into a single monolithic GPU execution step, yielding a 2-3x speedup over sequential execution.
• Microscaling (MX) Quantization: Full OCP-compliant MXFP4 implementation featuring block-wise E8M0 scale factors.
• Fully Quantized Training (FQT): Out-of-the-box FP8/INT8 backward pass compatibility utilizing Outlier Isolation (IQR 3.5) to mitigate quantization error loss.

Architecture Matrix & Evaluation

Structural Comparison

Local Benchmark Results (100 Steps, CPU)

Engineering Note: SDLA introduces higher initial computational overhead per step due to its recurrent state tracking, differential calculations, and token routing mechanics. However, it swaps quadratic runtime penalties for strict $O(N)$ scaling, unlocking massive sequence throughput at ultra-long context boundaries where standard MLA degrades.

Quick Start

1. Environment Setup

# Install package in editable/development mode
pip install -e .

# Run the comparative training script (SDLA vs MLA baseline)
python scripts/train.py

# Verify implementation integrity
python tests/test_import.py