transformers-v9 0.1.0

V9 Palace-structured attention for Transformers — block-diagonal palace-collated reasoning, per-palace LoRA, cross-palace gating

transformers-v9

V9 Palace-structured attention for HuggingFace Transformers — block-diagonal palace-collated reasoning, per-palace LoRA adapters, cross-palace gating.

Install

pip install transformers-v9

Quick Start

import transformers_v9  # registers v9_palace with Auto Classes

from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("v9_palace", vocab_size=32000, hidden_size=4096)
model = AutoModelForCausalLM.from_pretrained(config)

Architecture

• PalaceAttentionLayer: Low-rank QKV (r=64) + Palace block-diagonal mask + per-palace LoRA K/V (r=9, 38 palaces) + cross-palace gating
• 6 residual layers (stacked on frozen base LLM), no FFN — base model provides it
• ~50.4M trainable params at hidden_size=4096

Training

from transformers_v9 import V9PalaceTrainer

trainer = V9PalaceTrainer(model, train_dataset, eval_dataset)
trainer.train()

Requirements

• Python 3.8+
• PyTorch 2.0+
• transformers 4.30+
• accelerate 0.20+