KOLMGformers v0.0.3

Unified Python library merging two research-grade model families:

What's New in v0.0.3 — Bug Fixes & Improvements

Bug Fixes (KOLMOG)

Bug Fixes (Training)

New Features (v0.0.3)

KOLMOGformerConfig additions:

• context_dropout — independent dropout on the context vector path (default 0.0).
• ffn_type — FFN activation: "gelu" (default) | "silu" | "swiglu".
• max_position_embeddings_dynamic — RoPE cache auto-extends beyond limit instead of erroring (default True).
• validate() — called in __post_init__; surfaces config errors early with helpful messages.
• __repr__ — readable summary of key config fields.

TrainingArguments additions:

• early_stopping_patience — stop after N evaluations without improvement.
• gradient_checkpointing — enable memory-efficient training automatically.

Installation

pip install -e .
# Optional extras
pip install -e ".[hf]"       # HuggingFace tokenizers
pip install -e ".[flash]"    # Flash Attention 2
pip install -e ".[all]"      # Everything

Quick Start

KOLMOG — Attention-Free Causal LM

from kolmgformers import KOLMOGformerConfig, KOLMOGformerForCausalLM
import torch

config = KOLMOGformerConfig(
    vocab_size=32000,
    hidden_size=512,
    num_channels=8,
    num_layers=6,
    causal=True,
    use_nce=True,    # Normalized Context Extraction
    use_wcc=True,    # Weighted Cumulative Context (v0.0.2+)
    use_plkan=True,  # Piecewise Linear KAN — 3-5x faster (v0.0.2+)
)
model = KOLMOGformerForCausalLM(config)
print(config)  # v0.0.3: readable repr

ids = torch.tensor([[1, 42, 100]])
out = model.generate(ids, max_new_tokens=50, temperature=0.8)

KOLMOG — Training with Early Stopping

from kolmgformers import (
    KOLMOGTrainer, KOLMOGTrainingArguments,
    KOLMOGDataCollatorForCausalLM,
)

args = KOLMOGTrainingArguments(
    output_dir="runs/my_run",
    num_train_epochs=10,
    early_stopping_patience=3,   # v0.0.3: stop after 3 bad evals
    gradient_checkpointing=True, # v0.0.3: save memory on long seqs
    evaluation_strategy="steps",
    eval_steps=500,
)
trainer = KOLMOGTrainer(
    model=model,
    args=args,
    train_dataset=train_ds,
    eval_dataset=val_ds,
    data_collator=KOLMOGDataCollatorForCausalLM(pad_token_id=0),
)
trainer.train()

KOLMOG — True Checkpoint Resume

# v0.0.3: optimizer + scheduler state is saved, enabling true resume
trainer.load_checkpoint("runs/my_run/checkpoint-5000")
trainer.train()  # continues from exact state

OMG — Diffusion LM

from kolmgformers import OMGConfig, OMGModel

config = OMGConfig(vocab_size=32000, hidden_size=768, num_layers=12)
model  = OMGModel(config)

import torch
prompt = torch.tensor([[1, 42]])
out = model.generate(prompt, new_tokens=128, steps=10)

Architecture: KOLMOG

Based on the Kolmogorov-Arnold representation theorem:

F(X) = Σ_q Φ_q( Σ_i φ_{q,i}( xᵢ ⊕ eᵢ ⊕ c_{q,i} ) )

Key innovations:

• KAN layers — learnable B-spline activations per edge (not fixed non-linearities)
• NCE — Normalized Context Extraction: jackknife leave-one-out mean context
• WCC — Weighted Cumulative Context: attention-like token selectivity at O(n·d)
• PLKAN — Piecewise Linear KAN: 3–5× faster than B-spline, same expressivity
• No attention — O(n·d) time, O(d) memory (independent of sequence length)

Architecture: OMG

Parallel Diffusion Language Model with:

• GQA / MLA / Sliding-Window / Linear / Block-Sparse attention
• MoE (dense + soft MoE)
• DS-PDLM dual-stream (understanding + generation)
• LoRA / DoRA PEFT
• TASA + MFS + DI efficiency trilogy
• TWE temporal embeddings, NCA neuro-creative routing

Bug Fix History

License

Apache-2.0