Flatbuild

Build conversational language models from scratch.

An open-source framework for training compact decoder-only language models from random initialization using a modular, configuration-driven pipeline.

GitHub: https://github.com/flatseek/flatbuild  ·  Organization: https://github.com/flatseek

Part of the Flatseek ecosystem

Flatseek (Keyword Search) • Flatvec (Vector Search) • Flatask (RAG Runtime) • Flatlens (Data Visualization) • Flatbuild (LLM Training) • Flattune (LLM Fine-Tuning) • Flatrun (LLM Inference)

See It In Action

The repository includes a small conversational dataset that demonstrates the complete Flatbuild workflow—from dataset to training, export, and inference.

Example dataset (data/demo_chat/dataset.jsonl):

{"messages":[{"role":"system","content":"You are Flatbot, a friendly conversational assistant. You help users understand things, solve problems, and have natural conversations. Be clear, helpful, and concise."},{"role":"user","content":"What gas do plants absorb?"},{"role":"assistant","content":"Plants absorb carbon dioxide from the atmosphere."}]}

Train the model:

flatbuild train configs/demo_chat.yaml

Example output:

Loaded config: configs/demo_chat.yaml
Project: demo-chat
Loaded 2500 conversations
Train: 2375  Validation: 125

Epoch 1/4  loss=4.9630  val_loss=5.2337  ppl=187.49  acc=36.4%
Epoch 2/4  loss=4.2745  val_loss=4.0206  ppl=55.73   acc=46.3%
Epoch 3/4  loss=2.6530  val_loss=3.2722  ppl=26.37   acc=49.6%
Epoch 4/4  loss=1.9804  val_loss=2.8162  ppl=16.71   acc=54.2%

Run completed.

Artifacts:
outputs/demo/20260802T014922Z

Export the trained checkpoint:

flatbuild export \
    outputs/demo/20260802T014922Z/checkpoints/final \
    --format safetensors

Wrote model.safetensors
Copied tokenizer files

outputs/demo/20260802T014922Z/checkpoints/export_safetensors

Run the exported model with Flatrun.

pip install flatrun

Then run the exported checkpoint:

flatrun \
    --model outputs/demo-large/20260802T014922Z/checkpoints/export_safetensors \
    --prompt "who are you?"

Example output:

Detected format: safetensors
Tokenizer vocab: 2603
Chat template: Qwen2 ChatML
Loaded model in 0.01 s

Prompt:
who are you?

Generated:
'I I I 1 1 1 Ocean classic...... c solar that that that'

This tiny demonstration trains a decoder-only language model from scratch using 1,000 conversational examples and showcases the complete Flat pipeline:

Conversation Dataset
        │
        ▼
 Flatbuild Training
        │
        ▼
   Model Checkpoint
        │
        ▼
 SafeTensors Export
        │
        ▼
 Flatrun Inference

The bundled demo model is intentionally small and only trained for a few epochs, so the generated text is not expected to be meaningful. Its purpose is to demonstrate the complete end-to-end workflow. The same pipeline scales to larger datasets and larger language models without changing the training or deployment process.

Overview

A modular training framework for building compact language models.

Flatbuild is an open-source Python framework for training decoder-only language models entirely from random initialization.

Instead of requiring a large pre-trained checkpoint, Flatbuild provides every component of the training pipeline—including datasets, tokenizer training, model architecture, optimization, checkpointing, evaluation, and exporting—inside a single configurable framework.

Every training run is driven by a YAML configuration, making experiments reproducible, portable, and easy to automate.

Why Flatbuild?

Most modern training frameworks assume you already have a base model.

Flatbuild starts one layer lower.

It provides a complete reference implementation of the entire language-model training pipeline where every module can be inspected, replaced, extended, or optimized independently.

The resulting checkpoints can be exported to SafeTensors or HuggingFace Transformers and used directly by Flatrun or any compatible inference runtime.

Highlights

• Train decoder-only Transformers from scratch
• YAML-driven reproducible experiments
• Built-in tokenizer training
• Grouped Query Attention (GQA)
• RoPE positional embeddings
• RMSNorm and SwiGLU
• Gradient accumulation
• Mixed precision (FP32 / FP16 / BF16)
• Configurable chat templates
• JSONL, Parquet, and HuggingFace datasets
• SafeTensors and HuggingFace export
• Modular architecture for research and experimentation

Installation

pip install -e ".[dev]"

Development dependencies include:

• pytest
• ruff
• mypy

PyTorch is required at runtime.

Install the build matching your platform (CPU, CUDA, or Apple MPS).

Configuration

Training is controlled entirely through a YAML file.

Example:

name: demo-chat

dataset:
  type: conversation
  path: data/demo_chat/dataset.jsonl
  max_length: 384

tokenizer:
  source: train
  vocab_size: 1024

model:
  hidden_dim: 256
  n_layers: 4
  n_heads: 4
  n_kv_heads: 2
  context_length: 256

optimizer:
  type: adamw
  lr: 3e-4

trainer:
  epochs: 1
  batch_size: 8

See configs/demo_chat.yaml for the complete configuration.

CLI

flatbuild train
flatbuild resume
flatbuild evaluate
flatbuild export
flatbuild generate
flatbuild inspect
flatbuild benchmark

Run:

flatbuild --help

for the complete command reference.

Training Pipeline

Dataset
    │
    ▼
Tokenizer
    │
    ▼
Model
    │
    ▼
Training Loop
    │
    ▼
Checkpoint
    │
    ▼
Evaluation
    │
    ▼
Export

Every stage is configurable and replaceable.

Current Features

Flatbuild currently supports:

• Decoder-only Transformer
• Grouped Query Attention (GQA)
• Rotary Position Embeddings (RoPE)
• RMSNorm
• SwiGLU
• AdamW optimizer
• Cosine learning-rate scheduler
• Linear warmup
• Early stopping
• Gradient accumulation
• Mixed precision training
• BPE tokenizer training
• Chat template formatting
• JSONL datasets
• Parquet datasets
• HuggingFace datasets
• SafeTensors export
• HuggingFace Transformers export

Roadmap

Future releases are planned to include:

• Supervised Fine-Tuning (SFT)
• LoRA
• QLoRA
• DPO
• MoE architectures
• Vision-language models
• Distributed training
• Multi-node training
• Advanced evaluation benchmarks

Quick Start

Train the bundled demo model:

flatbuild train configs/demo_chat.yaml

A typical run will:

• Load the demo conversational dataset
• Train a tokenizer (or reuse an existing one)
• Train a decoder-only Transformer from scratch
• Save checkpoints, metrics, and metadata
• Export the final model

Outputs are written to:

outputs/demo/<run-id>/

License

Apache License 2.0

See LICENSE.