olaverse-foundry

A general-purpose toolkit for building transformer models — decoder or encoder.

olaverse-foundry is the model-building layer of the Olaverse ecosystem. Where olaverse gives you ready-to-use models, foundry lets you build new ones — pretraining, distilling, growing, adding heads, quantizing, and evaluating them. It is model-agnostic: any HuggingFace model or your own nn.Module works.

pretrain / distil → grow → add heads → quantize → evaluate → serve

Install

# Core (schema validation, growth planning — no GPU required)
pip install olaverse-foundry

# GPU training
pip install olaverse-foundry[torch]

# LoRA skill packs
pip install olaverse-foundry[torch,lego]

# Everything
pip install olaverse-foundry[all]

Quick start — embedding distillation (200M student)

from foundry import DataPipeline, EmbeddingDistillTrainer, EmbeddingDistillConfig
from transformers import AutoModel, AutoTokenizer

# Load student and teacher
student = AutoModel.from_pretrained("microsoft/deberta-v3-base")
teacher = AutoModel.from_pretrained("BAAI/bge-large-en-v1.5")
tok     = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base")

# Stream data
pipe = DataPipeline(
    source       = my_hf_dataset,
    tokenizer    = tok,
    batch_size   = 32,
    max_length   = 128,
    mode         = "embed",
    shuffle_buffer = 10_000,
)

# Train
trainer = EmbeddingDistillTrainer(
    student = student,
    teacher = teacher,
    config  = EmbeddingDistillConfig(
        loss         = "cosine",
        pool         = "mean",
        epochs       = 3,
        lr_scheduler = "cosine",
        warmup_steps = 200,
        torch_dtype  = "bfloat16",
        save_every   = 1000,
        save_dir     = "/checkpoints/embed-200m",
        log_backend  = "wandb",
    ),
)

result = trainer.train(pipe, eval_dataset=eval_pipe)
print(result["eval_losses"])

Quick start — causal LM distillation with multiple teachers

from foundry import (
    DataPipeline, TorchDistillTrainer, TorchTrainConfig,
    TeacherRegistry, FoundryRecipe,
)

# Build a registry of teachers
teachers = TeacherRegistry.from_names(
    ["meta-llama/Llama-3.1-70B", "Qwen/Qwen2-72B-Instruct"],
    weights=[1.0, 0.8],
)
teachers.load_all()

# Stream training data
pipe = DataPipeline(
    source     = my_dataset,
    tokenizer  = tok,
    batch_size = 8,
    max_length = 2048,
    mode       = "lm",
)

trainer = TorchDistillTrainer(
    student  = my_3b_model,
    teachers = teachers,
    config   = TorchTrainConfig(
        epochs                = 1,
        lr_scheduler          = "cosine",
        warmup_steps          = 500,
        torch_dtype           = "bfloat16",
        grad_accumulation_steps = 8,
        save_every            = 500,
        save_dir              = "/checkpoints/run1",
        eval_every            = 100,
        log_backend           = "wandb",
    ),
)

result = trainer.train(pipe, eval_dataset=eval_pipe)

Key components

Training features

All trainers share the same production-ready feature set:

• Mixed precision — torch_dtype="bfloat16" or "float16"
• Gradient accumulation — grad_accumulation_steps=N
• LR scheduler — "cosine" / "linear" / "constant" with linear warmup
• Reproducibility — seed=42 sets torch + numpy + random before training
• Checkpointing — save_checkpoint(path) / resume_from_checkpoint(path)
• Auto-checkpoint — save_every=N, save_dir="/path" saves every N steps
• Eval loop — eval_every=N evaluates on a held-out set every N steps
• W&B / TensorBoard — log_backend="wandb" or "tensorboard"
• OOM handling — CUDA OOM raises with actionable suggestions
• Streaming datasets — DataPipeline wraps any HF IterableDataset
• Dataset shuffling — shuffle=True or shuffle_buffer=N for streaming

CLI

# Check your environment
foundry doctor

# Preview a recipe plan (no GPU spend)
foundry plan recipe.yaml

# Run a recipe
foundry run recipe.yaml

# Run an embedding distillation recipe
foundry embed recipe.yaml

# List fusion strategies
foundry strategies

Recipe YAML

# recipe.yaml — full causal-LM factory
seed:
  model: meta-llama/Llama-3.1-8B
  init: pretrained

grow:
  method: depth_upscale
  to_params: 15B

teachers:
  - role: reasoning
    model: meta-llama/Llama-3.1-70B
    weight: 1.0

fusion:
  strategy: min_ce
  align: min_ed
  cache: topk_64

heal:
  tokens: 100B
  alpha: 0.3

output:
  freeze_base: true
  skillpacks: [ola_math, ola_code]

Optional extras

Documentation

Full docs: olaverse-labs.github.io/olaverse-foundry (auto-deployed from main).

Build or preview the site locally:

pip install -e ".[docs]"
mkdocs serve            # live preview at http://127.0.0.1:8000
mkdocs build --strict   # validate (no broken links / nav)

Links

• Main SDK — olaverse — ready-to-use models
• Homepage — olaverse.co.uk
• GitHub — Olaverse-Labs/olaverse-foundry
• Issues — GitHub Issues

License

Apache 2.0 — see LICENSE.