💀 NecroGraft: Pure W-Expansion

Full-Rank Weight Growth for Zero-Forgetting Model Adaptation

NecroGraft is not a fine-tuning patch. It is a new paradigm for how models acquire knowledge: not by being rewritten, but by growing.

Unlike Low-Rank Adaptation (LoRA), which approximates weight updates in a compressed subspace and permanently overwrites the base model (W_f + BA), Pure W-Expansion augments frozen pretrained models with full-rank trainable expansion matrices.

By strictly isolating gradient flow, NecroGraft achieves zero catastrophic forgetting by architectural guarantee, while matching or exceeding LoRA's domain learning capacity.

🧠 Core Philosophy: Growth vs. Rewrite

If a biological organism learns a new skill, it doesn't delete its motor cortex; it grows new synaptic pathways. NecroGraft applies this to neural networks. The frozen base weights W_f encode everything the model already knows. A freshly initialized, full-rank expansion matrix W_g learns the new domain. After training, a single concatenation merges them into one weight matrix indistinguishable from a natively trained model at inference.

🚀 Key Features

• 🛡️ Zero Catastrophic Forgetting: Base weights are mathematically isolated. They are never modified, never overwritten, and preserved bit-for-bit.
• 💪 Full-Rank Capacity: No low-rank bottlenecks. W_g is trained at full rank, giving it genuine capacity for complex, out-of-distribution domain representation.
• 🔥 The Yaka Gate ($\lambda$) Firewall: A learnable per-layer scalar gate that organically learns to amplify the graft in mid-depth layers and clamp it down in output-adjacent layers to protect the final distribution. No MoE routing, no auxiliary load-balancing losses.
• ⚡ Zero-Overhead Inference: After training, the Yaka Gate and RMS scaling are absorbed. The model collapses into a single standard nn.Linear matrix. No adapters, no extra FLOPs.
• 🎯 Upper-Layer Grafting Strategy: To bound computational cost and prevent input dimension growth from propagating through the entire network, NecroGraft strategically targets only the upper 6 layers (the final reasoning layers). This localizes the expansion, keeping inference latency identical to the base model while delivering massive domain adaptation.

📦 Installation

Make NecroGraft a first-class citizen in your ML stack:

pip install necrograft

📊 Empirical Results: NecroGraft vs. LoRA

We evaluated NecroGraft across multiple model families and extreme domain shifts. In every scenario, NecroGraft's full-rank expansion crushes LoRA's low-rank approximations.

1. DeepSeek-Coder-1.3B-Instruct (Multi-Domain)

Training: Upper 6 layers, 200 MMLU clinical samples, 2 epochs.

NecroGraft achieves lower perplexity on the training domain (MMLU) and ARC, while preserving base weights exactly.

2. Qwen2.5-1.5B-Instruct (Architecture-Agnostic Validation)

Training: Upper 6 layers, 200 samples, 2 epochs. Extreme out-of-domain shift.

🤯 The Qwen Breakthrough: When a model is completely lost in a new domain (Base PPL = 2502), LoRA's low-rank bottleneck struggles to map the new knowledge. NecroGraft drops the perplexity by 680× (2502 → 3.68), achieving a 29× improvement over LoRA on MMLU. This proves that for extreme domain shifts, full-rank geometric expansion is the only way to learn effectively without destroying the base.

🛠️ Quick Start

Wrapping a Hugging Face Model

NecroGraft provides a drop-in replacement for nn.Linear.

import torch
from transformers import AutoModelForCausalLM
from necrograft import NecroGraftLinear, apply_necrograft

# Load your base model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")

# Apply NecroGraft to the upper 6 layers (all 6 projections)
# Base weights are automatically frozen. Yaka Gate (λ) is initialized to 1.0.
apply_necrograft(
    model, 
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    expansion_dim=128,
    num_upper_layers=6
)

# Now train normally! Only W_g, W_proj, and λ are trainable.
# The base model's knowledge is structurally protected.

The combine_expansions Operation (Zero-Interference Theorem)

Train multiple domain grafts independently, then merge them into a single weight matrix with mathematically guaranteed zero cross-domain contamination.

from necrograft import combine_expansions

# Merge the medical graft and the coding graft into the base model
# No routing required. No expert collapse. Pure structural orthogonality.
merged_model = combine_expansions(
    base_model=model,
    expansions=[medical_graft, coding_graft],
    variant="learned_static" # Collapses to a single nn.Linear for zero overhead
)

📐 Mathematical Foundation

The Forward Pass

During training, the forward pass is defined as: $$ x_g = W_{proj} x $$ $$ y = W_f x + \lambda \cdot (W_g x_g) $$

Where:

• $W_f$ is the frozen pretrained weight matrix.
• $W_g$ is the trainable full-rank expansion matrix.
• $\lambda$ is the Yaka Gate, a learnable scalar initialized to 1.0.

The Zero-Interference Theorem

Given $n$ independently trained expansions, the combined weight matrix $W_{combined} = [W_f | W_{g1} | ... | W_{gn}]$ guarantees that for any domain $i$, the output is identical to the individually-merged model, up to the additive contributions of other domains. Cross-domain contributions cannot arise because the column-row correspondence of the block matrix multiplication is bijective per domain. Orthogonality is architecturally enforced, not statistical.

📜 Citation

If you use NecroGraft or Pure W-Expansion in your research, please cite the foundational paper:

@misc{yaka2026necrograft,
      title={Pure W-Expansion / NecroGraft: Full-Rank Weight Growth for Zero-Forgetting Model Adaptation}, 
      author={Heylel Yaka},
      year={2026},
      eprint={XXXXXXX},
      archivePrefix={Zenodo},
      primaryClass={cs.LG}
}

🧛‍♂️ Acknowledgements

Built by Heylel Yaka (Eric Heylel Danjuma Yaka)
Independent Researcher / Programmer, Abuja, Nigeria

Forged in the Grimoire of Elbàlor by The Digital Necromancer. 💀🔥

Special thanks to the open-source ML community. We do not overwrite. We grow.