transformers-v9 0.2.0

V9 Palace Transformer — from-scratch LLM with block-diagonal palace-collated attention, per-palace LoRA K/V, cross-palace gating, and semantic palace routing

transformers-v9

V9 Palace Transformer — an independent from-scratch LLM architecture with block-diagonal palace-collated attention, per-palace LoRA K/V adapters, and cross-palace semantic gating. Not a base-model adapter — its own embedding/position/lm_head stack.

Part of the V∞ (SiliconLifeOS) paradigm: cognitive structure is architecture, not prompt engineering.

Architecture

Component	Detail
Palace Attention	Block-diagonal mask: tokens attend only within same palace
LoRA K/V	Per-palace low-rank adapters (r=16, 48 palaces)
Cross-palace gate	Softmax MLP mix of palace embeddings
QKV low-rank	D→64→D bottleneck projections
Residuals	6 layers, no FFN (sufficient for structured reasoning)
Default scale	~1.5B params (hidden=2048, 6 layers, 48 palaces)

Input → Embed/Pos → 6×PalaceAttentionLayer (residual) → LM Head → Output

Quick Start

import transformers_v9  # registers v9_palace with HF Auto Classes

from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("v9_palace", vocab_size=32768, hidden_size=2048)
model = AutoModelForCausalLM.from_config(config)

Training

from transformers_v9 import V9PalaceTrainer

trainer = V9PalaceTrainer(model, train_dataset, eval_dataset)
trainer.train()

From-scratch training pipeline included (from transformers_v9.train import get_curriculum_dataset):

• SPC L1-L6 curriculum data
• CP fusion tasks
• Multi-domain reasoning

Router (Semantic Palace Routing)

from transformers_v9 import V9RouterConfig, V9Router

router = V9Router(V9RouterConfig())
palace = router.route("Discuss the attention mechanism in Transformers")
# → Palace ID for attention-related reasoning

4-level cascade: L0 domain hash → L1 keyword voting → L2 group Jaccard → L3 embedding (optional).

Install

pip install transformers-v9

Requirements

• Python 3.8+
• PyTorch 2.0+
• transformers 4.30+
• accelerate 0.20+

Status

Alpha prototype — structured reasoning validation complete. CPU training verified (500 samples, 20 epochs: ont_self 0.55→1.11, accuracy 96.5%). GPU training for full-scale (1.5B+) recommended.

Links

• V∞ Paradigm
• GitHub