language:

• en license: apache-2.0 library_name: sdory-dora tags:
• transformers
• peft
• dora
• recurrent-loop
• latent-recurrence
• zero-forgetting
• low-vram
• cybersecqwen
• qwen base_model: athena129/CyberSecQwen-4B pipeline_tag: text-generation inference: true

SDory-DoRA: Sparse Latent-Recurrent Decomposed Architecture

SDory-DoRA is a next-generation parameter-efficient architecture that fuses NVIDIA Dory's Latent-Recurrent Looping Core with Weight-Decomposed Low-Rank Adaptation (DoRA), Top-5% Sparsity Filtering, and Orthogonal Anti-Forgetting Constraints.

Designed specifically for consumer hardware (e.g., laptop GPUs with 8 GB VRAM), SDory-DoRA allows training and deploying frontier-grade reasoning models using only ~2.7 GB of GPU VRAM and producing micro-skill checkpoints under 4 Megabytes.

Key Breakthroughs & Capabilities

Architectural Blueprint

                              User Prompt
                                   │
                                   ▼
        ┌───────────────────────────────────────────────────────┐
        │ 1. ZERO-RAM DIRECT STREAMER (Direct-to-VRAM)          │
        │    • Direct-to-GPU memory mapping (0 MB RAM leak)     │
        └──────────────────────────┬────────────────────────────┘
                                   │
                                   ▼
        ┌───────────────────────────────────────────────────────┐
        │ 2. INPUT ANCHOR BLOCK (Layers 0 to 7)                 │
        │    • Encodes prompt tokens & establishes context      │
        └──────────────────────────┬────────────────────────────┘
                                   │
                                   ▼
  ╔═══════════════════════════════════════════════════════════════════╗
  ║ 3. RECURRENT DORY REASONING CORE (20 Shared Middle Layers)        ║
  ║    (Executes K=1 to K=4 Thinking Loops on the same weights)       ║
  ║                                                                   ║
  ║    ┌─────────────────────────────────────────────────────────┐    ║
  ║    │ SPARSITY FILTER (Top 5% Expert Channels)                │    ║
  ║    │ • Prunes 95% inactive neurons; active size < 4 MB       │    ║
  ║    └───────────────────────────┬─────────────────────────────┘    ║
  ║                                │                                  ║
  ║    ┌───────────────────────────▼─────────────────────────────┐    ║
  ║    │ DoRA ENGINE (Decoupled Magnitude + Direction)           │    ║
  ║    │ • Magnitude (m) in FP32 preserves unblurred clarity     │    ║
  ║    └───────────────────────────┬─────────────────────────────┘    ║
  ║                                │                                  ║
  ║    ┌───────────────────────────▼─────────────────────────────┐    ║
  ║    │ ORTHOGONAL NULL-SPACE ANCHOR                            │    ║
  ║    │ • Enforces W_new ⊥ W_base = 0 (Zero Forgetting)         │    ║
  ║    └─────────────────────────────────────────────────────────┘    ║
  ║                                                                   ║
  ║    ↺ Thinking Loop 1: Syntax & Vulnerability Identification       ║
  ║    ↺ Thinking Loop 2: Exploit Analysis & Proof-of-Concept         ║
  ║    ↺ Thinking Loop 3: Patch Generation & Verification             ║
  ║                                                                   ║
  ║    [Last-Loop KV Cache]: Only caches final loop -> Flat VRAM      ║
  ╚═══════════════════════════════════════════════════════════════════╝
                                   │
                                   ▼
        ┌───────────────────────────────────────────────────────┐
        │ 4. OUTPUT BLOCK (Layers 28 to 35) + FACTUAL GUARD     │
        │    • Compares generated answer against ground truth   │
        └──────────────────────────┬────────────────────────────┘
                                   │
                                   ▼
                  CRYSTAL CLEAR, 100% FACTUAL ANSWER

Quickstart Installation

Install the library directly using pip:

git clone https://github.com/vantix-ai/sdory-dora.git
cd sdory-dora
pip install -e .

Python Usage Example

1. Training a Custom Skill in 3 Lines of Code

from transformers import AutoTokenizer
from sdory_dora import SDoryDoRAConfig, SDoryDoRAForCausalLM, SDoryDoRATrainer

# Initialize architecture configuration
config = SDoryDoRAConfig(
    default_recurrent_loops=2,  # 2 Thinking loops
    dora_rank=8,
    sparsity_ratio=0.05        # Top 5% expert neurons
)

# Load model (automatically configures 4-bit NF4 base + recurrent core)
model = SDoryDoRAForCausalLM(config=config)
tokenizer = AutoTokenizer.from_pretrained(config.model_name_or_path, trust_remote_code=True)

# Train on any JSONL dataset
trainer = SDoryDoRATrainer(model=model, tokenizer=tokenizer, batch_size=1)
trainer.train(
    data_path="my_dataset.jsonl",
    epochs=2,
    output_dir="checkpoints/my_new_skill"
)
# Result: Saves a 3.37 MB micro-adapter!

2. High-Performance Inference with Elastic Thinking Depth

from transformers import AutoTokenizer
from sdory_dora import SDoryDoRAConfig, SDoryDoRAForCausalLM, SDoryDoRAGenerator

config = SDoryDoRAConfig(default_recurrent_loops=2)
model = SDoryDoRAForCausalLM(config=config)

# Hot-swap the trained 3.37 MB micro-adapter in < 0.1s:
model.load_micro_adapter("checkpoints/my_new_skill/trained_sdory_dora_skill.pt")
tokenizer = AutoTokenizer.from_pretrained(config.model_name_or_path, trust_remote_code=True)

generator = SDoryDoRAGenerator(model=model, tokenizer=tokenizer)

# Fast Mode (1 Loop) vs Deep Reasoning Mode (3 Loops)
response = generator.generate(
    prompt="Explain Kerberos Golden Ticket attack and prevention.",
    loops=3,  # Elastic Thinking Mode
    max_new_tokens=128
)

print(response["generated_text"])
print("Grounding Confidence:", response["confidence"])

Command-Line Interface (CLI)

Train or chat directly from your terminal:

# 1. Train on your data
sdory-train --data "dataset.jsonl" --epochs 2 --loops 2 --output_dir "checkpoints/my_skill"

# 2. Interactive chat with elastic thinking loops
sdory-chat --adapter "checkpoints/my_skill/trained_sdory_dora_skill.pt" --loops 2

Hardware Safety & Compatibility

• Validated Hardware: NVIDIA GeForce RTX 5050 Laptop GPU (8 GB VRAM), 16 GB RAM.
• Operating System: Windows 11 / Linux.
• VRAM Ceiling: Hard-capped at 80% allocation fraction to guarantee dedicated OS display headroom.
• RAM Optimization: Direct-to-VRAM disk streaming with Windows EmptyWorkingSet() cache purging.

Citation

@software{vantix_sdory_dora_2026,
  author = {VANTiX Labs},
  title = {SDory-DoRA: Sparse Latent-Recurrent Decomposed Architecture for Zero-Forgetting LLM Adaptation},
  year = {2026},
  url = {https://github.com/vantix-ai/sdory-dora}
}