Frankenstein Transformer

See frankenstein-transformer for a web interface to configure your YAML!

Config-driven transformer experimentation toolkit with 36 mixer architectures and 23 optimizer families.

Quick Start

Verify: frankenstein-transformer --help

Feature Matrix

Architecture Decision Table

See configs/README.md for preset details and docs/specs/ for architecture deep-dives.

CLI Command Reference

All model-executing commands accept --device auto|cpu|cuda|mps.

Mixer Categories

⚠️ sparge_attn and fasa_attn are eval-only — training raises a runtime error.

Configure via layer_pattern in YAML. See src/schema.yaml for the full mixer reference table.

Optimizer Categories

Parameters use prefixed keys: <optimizer_class>-<group>_<param> (e.g. adamw-lr_embeddings, muon-ns_steps). See configs/README.md for the full parameter reference.

Documentation Map

Installation

uv (recommended)

git clone https://github.com/erickfmm/frankenstein-transformer.git
cd frankenstein-transformer
uv venv
source .venv/bin/activate
uv pip install -e ".[train]"

pip

python -m venv .venv
source .venv/bin/activate
pip install -e ".[train]"

conda

conda create -n frankenstein python=3.9
conda activate frankenstein
pip install -e ".[train]"

Verify installation:

frankenstein-transformer --help

Quick Training Example

Minimal YAML config (my_config.yaml) — only the 5 required model fields plus task; everything else uses FrankensteinModelConfig/TrainingConfig defaults:

model_class: frankenstein
model:
  vocab_size: 30522
  hidden_size: 256
  num_layers: 4
  num_heads: 8
  layer_pattern: [standard_attn, standard_attn, standard_attn, standard_attn]
training:
  task: mlm
  batch_size: 8
  max_length: 128
  mlm_probability: 0.15
  max_samples: 100000
  dataset_batch_size: 10000
  num_workers: 4
  cache_dir: "./temp_data/cache"
  optimizer:
    optimizer_class: adamw
    parameters:
      adamw-lr_embeddings: 1e-4
      adamw-lr_norms: 1e-4
      adamw-lr_attention: 1e-4
      adamw-lr_other: 1e-4
      adamw-wd_embeddings: 0.01
      adamw-wd_norms: 0.01
      adamw-wd_attention: 0.01
      adamw-wd_other: 0.01
      adamw-betas_embeddings: [0.9, 0.95]
      adamw-betas_norms: [0.9, 0.95]
      adamw-betas_attention: [0.9, 0.95]
      adamw-betas_other: [0.9, 0.95]
      adamw-eps_embeddings: 1e-8
      adamw-eps_norms: 1e-8
      adamw-eps_attention: 1e-8
      adamw-eps_other: 1e-8
  scheduler_total_steps: 1000

Unspecified model fields fall back to FrankensteinModelConfig defaults (num_loops=2, dropout=0.1, norm_type=dynamic_tanh, use_moe=true, ffn_activation=silu, etc.). Unspecified training fields fall back to TrainingConfig defaults (scheduler_type=cosine, grad_clip_max_norm=5.0, gpu_temp_guard_enabled=true, etc.). Override only what you need to change.

Run:

frankenstein-transformer train --config my_config.yaml --device auto

List available named presets:

frankenstein-transformer train --list-configs

Vision Transformer (frankenstein_vit) Example

The frankenstein_vit model class (arXiv:2010.11929) splits images into patches, embeds them, and processes the sequence through the same HybridLayer stack as the text models. It supports three tasks: patch_prediction (autosupervised masked patch prediction), classification (image classification), and segmentation (per-pixel or EoMT query-based, arXiv:2503.19108).

Minimal classification YAML (vit_config.yaml):

model_class: frankenstein_vit
model:
  dims:
    hidden_size: 768
    num_layers: 12
    num_heads: 12
    layer_pattern: [standard_attn]
    mode: encoder
  norm: {type: layer_norm}
  use_moe: false
  use_bitnet: false
  ffn_activation: gelu
  ffn_hidden_size: 3072
image:
  image_size: {height: 224, width: 224}
  patch_size: 16
  in_channels: 3
  pos_embedding_type: learned_1d
  cls_token: true
  pooling_mode: cls
  num_classes: 10
dataset:
  dataset_name: cifar10
  rescale: {height: 224, width: 224}
training:
  task: classification
  batch_size: 512
  num_epochs: 90
  optimizer:
    optimizer_class: adamw
    parameters:
      adamw-lr_other: 0.001
      adamw-wd_other: 0.1
  classification:
    batch_size: 512
    num_epochs: 90
    learning_rate: 0.001

Run: frankenstein-transformer train --config vit_config.yaml --device auto

See configs/frankenstein_vit_base.yaml for the full ViT-Base/16 preset and docs/specs/vision.md for the complete specification.

License

Apache License 2.0 — see LICENSE for full text.