wiola 0.1.1

Wiola: a novel decoder-only Small Language Model with Spiral Rotary Positional Encoding, Gated Cross-Layer Attention, Adaptive Token Merging, Dual-Stream Feed-Forward, and WiolaRMSNorm.

Wiola

A novel decoder-only Small Language Model built from first principles.

Wiola is an experimental, fully open-source language model architecture that introduces five independently novel components. It is implemented in PyTorch, integrates natively with 🤗 Transformers (AutoModelForCausalLM via trust_remote_code=True), ships in four sizes (120M / 360M / 700M / 1.5B), and is covered by 22 unit tests.

Status — experimental research prototype. The architecture is implemented, tested, and ready to train. Pre-trained weights and benchmark numbers are future work; any perplexity/scaling figures in the paper are projections, not measurements. See Limitations.

---

Table of contents

• The five novel components
• Model family
• Installation
• Quickstart
• Training
• Evaluation
• Inference examples
• Publishing to the Hugging Face Hub
• Repository layout
• Testing
• Limitations
• Contributing
• License & citation

---

The five novel components

Component	What it does
SRPE — Spiral Rotary Positional Encoding	Places token positions on a 3D helical manifold with dual winding angles and a sinusoidal radial term, encoding multiple positional scales analytically (no extra parameters).
GCLA — Gated Cross-Layer Attention	GQA self-attention + SRPE, plus soft cross-attention to compressed summaries of preceding layers, blended by a learned scalar gate and a per-position sigmoid output gate.
ATM — Adaptive Token Merging	During training only, greedily merges semantically redundant adjacent tokens (cosine similarity > τ) in the middle third of layers, then exactly restores length. Disabled at inference.
DSFF — Dual-Stream Feed-Forward	Two parallel dense streams — a narrow SwiGLU stream and a wide GELU stream — fused by a learned per-dimension sigmoid gate.
WiolaRMSNorm	RMSNorm with a learned per-dimension offset δ added before normalisation; δ = 0 recovers standard RMSNorm exactly.

A full mathematical treatment is in docs/architecture.md and the accompanying paper.

---

Model family

Variant	d	Layers	Heads (H/H_kv)	Params	Config
wiola-120m	768	12	12 / 4	~120M	configs/wiola-120m.yaml
wiola-360m	1024	16	16 / 4	~360M	configs/wiola-360m.yaml
wiola-700m	1536	24	16 / 8	~700M	configs/wiola-700m.yaml
wiola-1.5b	2048	28	16 / 8	~1.5B	configs/wiola-1.5b.yaml

All variants use a 32,000-token byte-level BPE vocabulary and a 2,048-token context window.

---

Installation

Requires Python 3.9+ and a recent PyTorch. On Windows, install PyTorch from the official selector first if you need a specific CUDA build.

git clone https://github.com/Wiola-OSCOWL-ai/wiola.git
cd wiola

python -m venv .venv
# Windows (Git Bash):   source .venv/Scripts/activate
# Linux / macOS:        source .venv/bin/activate

pip install -e ".[train]"      # editable install + training extras

Verify the install:

python -c "from wiola import WiolaForCausalLM, WiolaConfig; \
print(WiolaForCausalLM(WiolaConfig.from_preset('wiola-120m')).num_parameters())"

---

Quickstart

Build a model from a preset and run a forward pass — no download required:

import torch
from wiola import WiolaConfig, WiolaForCausalLM

config = WiolaConfig.from_preset("wiola-120m")
model = WiolaForCausalLM(config).eval()

input_ids = torch.randint(0, config.vocab_size, (1, 16))
out = model(input_ids=input_ids)
print(out.logits.shape)          # torch.Size([1, 16, 32000])

generated = model.generate(input_ids, max_new_tokens=20, do_sample=False)
print(generated.shape)           # torch.Size([1, 36])

---

Training

The training pipeline has one source of truth for text: either a local corpus or a Hugging Face dataset (set exactly one in the config). If a tokenizer is missing, it is built automatically instead of crashing.

1. (Optional) Build the tokenizer ahead of time

python scripts/prepare_tokenizer.py --config configs/wiola-120m.yaml

This step is optional — train.py will build it on demand if it is absent.

2. (Optional) Tokenize and pack the corpus

python scripts/prepare_data.py --config configs/wiola-120m.yaml

3. Train

python scripts/train.py --config configs/wiola-120m.yaml

Resume from a checkpoint:

python scripts/train.py --config configs/wiola-120m.yaml --resume runs/wiola-120m/latest

Using your own text

Edit the data block of a config to point at local files and clear the HF fields:

data:
  local_path: ./data/my_corpus     # a file, a directory, or a glob
  hf_dataset: null
  hf_config: null
  block_size: 1024
  val_fraction: 0.01

What gets logged

Each run tracks training loss, validation loss, perplexity, learning rate, gradient norm, throughput (tokens/sec) and GPU memory. Set logging.backend to tensorboard or wandb to stream them, or leave it as none for stdout only. Checkpoints follow a latest / best / step-N / final lifecycle, and each one stores the run config and random seed for reproducibility. See docs/training.md for the full reference.

---

Evaluation

Compute perplexity on a held-out split:

python scripts/evaluate.py --model runs/wiola-120m/best --split test

The script also prints the exact lm-eval harness commands for HellaSwag, PIQA and ARC-Easy so you can reproduce standard benchmarks once you have trained weights. See docs/evaluation.md.

---

Inference examples

python examples/generate.py --model runs/wiola-120m/best --prompt "The history of"
python examples/chat.py     --model runs/wiola-120m/best
python examples/batch.py    --model runs/wiola-120m/best

Loading a published checkpoint from the Hub (note trust_remote_code=True, which is required because Wiola ships custom modeling code):

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "oscowlai/wiola-120m"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

---

Publishing to the Hugging Face Hub

After training, push weights + tokenizer + custom code + model card in one step:

huggingface-cli login
python scripts/push_to_hub.py \
    --model runs/wiola-120m/best \
    --repo oscowlai/wiola-120m \
    --card model_cards/wiola-120m.md

A complete, copy-paste GitHub + Hugging Face walkthrough for Windows 10 / VS Code / Git Bash lives in docs/publishing.md.

---

Repository layout

wiola/
├── src/wiola/
│   ├── configuration_wiola.py      # WiolaConfig (+ from_preset)
│   ├── modeling_wiola.py           # WiolaModel, WiolaForCausalLM
│   ├── components/                 # SRPE, GCLA, ATM, DSFF, WiolaRMSNorm
│   ├── data/                       # one-source-of-truth data pipeline
│   └── utils.py
├── configs/                        # _base.yaml + 4 variant configs
├── scripts/                        # prepare_tokenizer/data, train, evaluate, push_to_hub
├── examples/                       # generate, chat, batch
├── tests/                          # 22 unit tests
├── docs/                           # architecture, training, evaluation, publishing
├── model_cards/                    # one card per variant (for the Hub)
└── .github/workflows/ci.yml        # lint + tests

---

Testing

pip install -e ".[dev]"
pytest -q

All 22 tests should pass. They cover every component plus the assembled model, including the incremental-decoding correctness check (full forward vs. cached two-chunk forward agree to l_inf < 1e-4).

---

Limitations

• ATM is disabled at inference to keep the KV-cache consistent.
• GCLA's layer-to-layer dependency complicates pipeline parallelism.
• SRPE's radial term may exhibit phase interference for very long contexts.
• No pre-trained weights are released yet. Projected perplexity and scaling figures are hypotheses until models are trained and evaluated.

---

Contributing

Contributions are welcome — see CONTRIBUTING.md and the issue / pull-request templates under .github/. Please run ruff check . and pytest before opening a PR.

---

License & citation

Released under the Apache License 2.0.

@misc{wiola2025,
  title  = {The Wiola Architecture for Efficient Small Language Models},
  author = {Aryuemaan Chowdhury},
  year   = {2026},
  howpublished = {\url{https://github.com/Wiola-OSCOWL-ai/wiola}}
}